[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:34:57 dist-test-slave-rmhg krb5kdc[19834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:34:57 dist-test-slave-rmhg krb5kdc[19834](info): set up 2 sockets
Aug 09 04:34:57 dist-test-slave-rmhg krb5kdc[19834](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 04:34:58.380977 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:34:58 dist-test-slave-rmhg krb5kdc[19834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:34:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:34:58.508314 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:38901
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:34:58.620822 19852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:34:58.621091 19852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:34:58.621165 19852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:34:58.625051 19852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:34:58.625106 19852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:34:58.625133 19852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:34:58.625171 19852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:34:58.625200 19852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:34:58.630342 19852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:34:58.631394 19852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:34:58.632431 19852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:34:58.641271 19864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:34:58.641278 19861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:34:58.641278 19862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:34:58.641331 19852 server_base.cc:1034] running on GCE node
I20260809 04:34:58.642025 19852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:34:58.643132 19852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:34:58.644301 19852 hybrid_clock.cc:648] HybridClock initialized: now 1786250098644294 us; error 92 us; skew 500 ppm
Aug 09 04:34:58 dist-test-slave-rmhg krb5kdc[19834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250098, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:34:58.647113 19852 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:34:58.648059 19852 webserver.cc:460] Webserver started at http://127.19.92.62:33101/ using document root <none> and password file <none>
I20260809 04:34:58.648624 19852 fs_manager.cc:359] Metadata directory not provided
I20260809 04:34:58.648692 19852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:34:58.648903 19852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:34:58.651111 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "48e624eb7a9749278513b49a9c06f96b"
format_stamp: "Formatted at 2026-08-09 04:34:58 on dist-test-slave-rmhg"
server_key: "f724a1d29576301f320803547209abaf"
server_key_iv: "59a1e78deac21c0acd0c6e5ca10ff5e2"
server_key_version: "encryptionkey@0"
I20260809 04:34:58.651657 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "48e624eb7a9749278513b49a9c06f96b"
format_stamp: "Formatted at 2026-08-09 04:34:58 on dist-test-slave-rmhg"
server_key: "f724a1d29576301f320803547209abaf"
server_key_iv: "59a1e78deac21c0acd0c6e5ca10ff5e2"
server_key_version: "encryptionkey@0"
I20260809 04:34:58.655377 19852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 04:34:58.657788 19873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:58.659083 19852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:34:58.659215 19852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "48e624eb7a9749278513b49a9c06f96b"
format_stamp: "Formatted at 2026-08-09 04:34:58 on dist-test-slave-rmhg"
server_key: "f724a1d29576301f320803547209abaf"
server_key_iv: "59a1e78deac21c0acd0c6e5ca10ff5e2"
server_key_version: "encryptionkey@0"
I20260809 04:34:58.659315 19852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:34:58.670145 19852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:34:58.673105 19852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:34:58.673331 19852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:34:58.680055 19852 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:38901
I20260809 04:34:58.680106 19948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:38901 every 8 connection(s)
I20260809 04:34:58.680992 19852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:34:58.684091 19949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:34:58.684813 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 19852
I20260809 04:34:58.684995 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:34:58.686781 19824 external_mini_cluster.cc:1428] Setting key dd0e8bf8bf5c1a351822297e58238185
I20260809 04:34:58.691483 19949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b: Bootstrap starting.
I20260809 04:34:58.693795 19949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b: Neither blocks nor log segments found. Creating new log.
I20260809 04:34:58.694518 19949 log.cc:824] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:34:58.696367 19949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b: No bootstrap required, opened a new log
I20260809 04:34:58.699810 19949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e624eb7a9749278513b49a9c06f96b" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38901 } }
I20260809 04:34:58.700037 19949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:34:58.700094 19949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e624eb7a9749278513b49a9c06f96b, State: Initialized, Role: FOLLOWER
I20260809 04:34:58.700527 19949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e624eb7a9749278513b49a9c06f96b" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38901 } }
I20260809 04:34:58.700661 19949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:34:58.700727 19949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:34:58.700831 19949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:34:58.701949 19949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e624eb7a9749278513b49a9c06f96b" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38901 } }
I20260809 04:34:58.702270 19949 leader_election.cc:302] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48e624eb7a9749278513b49a9c06f96b; no voters:
I20260809 04:34:58.702545 19949 leader_election.cc:288] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:34:58.702684 19955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:34:58.702896 19955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [term 1 LEADER]: Becoming Leader. State: Replica: 48e624eb7a9749278513b49a9c06f96b, State: Running, Role: LEADER
I20260809 04:34:58.703312 19955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e624eb7a9749278513b49a9c06f96b" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38901 } }
I20260809 04:34:58.703699 19949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:34:58.705078 19958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48e624eb7a9749278513b49a9c06f96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e624eb7a9749278513b49a9c06f96b" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38901 } } }
I20260809 04:34:58.705253 19958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [sys.catalog]: This master's current role is: LEADER
Aug 09 04:34:58 dist-test-slave-rmhg krb5kdc[19834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:34:58.705572 19959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48e624eb7a9749278513b49a9c06f96b. Latest consensus state: current_term: 1 leader_uuid: "48e624eb7a9749278513b49a9c06f96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e624eb7a9749278513b49a9c06f96b" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38901 } } }
I20260809 04:34:58.705653 19959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b [sys.catalog]: This master's current role is: LEADER
W20260809 04:34:58.709340 19975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:34:58.709408 19975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:34:58.709478 19971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:34:58.710361 19971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:34:58.712648 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:58.690815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.19.92.62:38901)
0809 04:34:58.691334 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0809 04:34:58.691355 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:34:58.692381 (+ 1026us) server_negotiation.cc:407] Connection header received
0809 04:34:58.695703 (+ 3322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:34:58.695721 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:34:58.696042 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:34:58.696345 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:34:58.698552 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:58.699526 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:58.700787 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:58.701498 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:58.701670 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:58.706842 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:34:58.706849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:34:58.706863 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:34:58.706893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:34:58.708850 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:58.709592 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:58.709598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:58.709604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:58.709716 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:58.710140 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:58.710143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:58.710144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:58.710620 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:34:58.710831 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 04:34:58.711414 (+ 583us) server_negotiation.cc:299] Negotiation successful
0809 04:34:58.711700 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":163,"threads_started":1}
I20260809 04:34:58.715952 19971 catalog_manager.cc:1347] Generated new cluster ID: 7a525d86962246fd8dfac5bea755e5c9
I20260809 04:34:58.716010 19971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:34:58.729612 19971 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:34:58.731060 19971 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:34:58.737306 19971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48e624eb7a9749278513b49a9c06f96b: Generated new TSK 0
I20260809 04:34:58.738035 19971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:34:58.778549 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38901
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:34:58.909090 19981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:34:58.909366 19981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:34:58.909409 19981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:34:58.913331 19981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:34:58.913385 19981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:34:58.913443 19981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:34:58.918694 19981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:34:58.919754 19981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:34:58.920785 19981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:34:58.931541 19987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:34:58.931520 19991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:34:58.931528 19988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:34:58.931742 19981 server_base.cc:1034] running on GCE node
I20260809 04:34:58.932322 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:34:58.932961 19981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:34:58.934149 19981 hybrid_clock.cc:648] HybridClock initialized: now 1786250098934130 us; error 31 us; skew 500 ppm
Aug 09 04:34:58 dist-test-slave-rmhg krb5kdc[19834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250098, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:34:58.937269 19981 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:34:58.938129 19981 webserver.cc:460] Webserver started at http://127.19.92.1:39403/ using document root <none> and password file <none>
I20260809 04:34:58.938656 19981 fs_manager.cc:359] Metadata directory not provided
I20260809 04:34:58.938715 19981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:34:58.938882 19981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:34:58.940892 19981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "c6b9f152f3434c3885653060069e8cf0"
format_stamp: "Formatted at 2026-08-09 04:34:58 on dist-test-slave-rmhg"
server_key: "0086b53d6b73646d1ec9b18f111a6003"
server_key_iv: "fbb2ea9f42a9a2f554a75bc1479ad207"
server_key_version: "encryptionkey@0"
I20260809 04:34:58.941502 19981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "c6b9f152f3434c3885653060069e8cf0"
format_stamp: "Formatted at 2026-08-09 04:34:58 on dist-test-slave-rmhg"
server_key: "0086b53d6b73646d1ec9b18f111a6003"
server_key_iv: "fbb2ea9f42a9a2f554a75bc1479ad207"
server_key_version: "encryptionkey@0"
I20260809 04:34:58.945631 19981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:34:58.949860 20002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:58.951112 19981 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260809 04:34:58.951217 19981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "c6b9f152f3434c3885653060069e8cf0"
format_stamp: "Formatted at 2026-08-09 04:34:58 on dist-test-slave-rmhg"
server_key: "0086b53d6b73646d1ec9b18f111a6003"
server_key_iv: "fbb2ea9f42a9a2f554a75bc1479ad207"
server_key_version: "encryptionkey@0"
I20260809 04:34:58.951325 19981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:34:58.971369 19981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:34:58.974646 19981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:34:58.974829 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:34:58.975453 19981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:34:58.976303 19981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:34:58.976351 19981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:58.976392 19981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:34:58.976420 19981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:58.988787 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:37441
I20260809 04:34:58.988822 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:37441 every 8 connection(s)
I20260809 04:34:58.989749 19981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:34:58.996730 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 19981
I20260809 04:34:58.996822 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:34:58.997058 19824 external_mini_cluster.cc:1428] Setting key 2aac9f1741594e4734e39ba53b304a29
Aug 09 04:34:58 dist-test-slave-rmhg krb5kdc[19834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250098, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:34:59.004242 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:58.991663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:47721 (local address 127.19.92.62:38901)
0809 04:34:58.991822 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 04:34:58.991825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:34:58.992827 (+ 1002us) server_negotiation.cc:407] Connection header received
0809 04:34:58.993712 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:34:58.993714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:34:58.993764 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:34:58.993837 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:34:58.995164 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:58.995697 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:58.997084 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:58.997410 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:58.997509 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:59.000407 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:34:59.000412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:34:59.000415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:34:59.000468 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:34:59.001998 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.002499 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.002503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.002505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.002600 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.002931 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.002934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.002936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.003181 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:34:59.003280 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:34:59.003868 (+ 588us) server_negotiation.cc:299] Negotiation successful
0809 04:34:59.004058 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:34:59.005223 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:58.992011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38901 (local address 127.19.92.1:47721)
0809 04:34:58.992672 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0809 04:34:58.992721 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:34:58.993531 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:34:58.994045 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:34:58.994052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:34:58.994526 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:34:58.995037 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:34:58.995052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:34:58.995829 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:34:58.995833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:34:58.996942 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:34:58.996953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:34:58.997566 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:34:58.997570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:34:58.997780 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:58.998409 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:34:58.998454 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:34:59.000210 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:34:59.002120 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:34:59.002126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:34:59.002139 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:34:59.002401 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:34:59.002695 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:34:59.002697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:34:59.002698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:34:59.002846 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:34:59.003286 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:34:59.003292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:34:59.003614 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 04:34:59.003787 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 04:34:59.004131 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":106,"threads_started":1}
I20260809 04:34:59.006605 20173 heartbeater.cc:347] Connected to a master server at 127.19.92.62:38901
I20260809 04:34:59.006914 20173 heartbeater.cc:464] Registering TS with master...
I20260809 04:34:59.007598 20173 heartbeater.cc:511] Master 127.19.92.62:38901 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:34:59.009392 19901 ts_manager.cc:194] Registered new tserver with Master: c6b9f152f3434c3885653060069e8cf0 (127.19.92.1:37441)
I20260809 04:34:59.010568 19901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:47721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:34:59.027355 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38901
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:34:59.162042 20180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:34:59.162307 20180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:34:59.162350 20180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:34:59.166525 20180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:34:59.166603 20180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:34:59.166666 20180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:34:59.172549 20180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:34:59.173689 20180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:34:59.174690 20180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:34:59.184949 20193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:34:59.184978 20188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:34:59.184952 20191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:34:59.185169 20180 server_base.cc:1034] running on GCE node
I20260809 04:34:59.185736 20180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:34:59.186349 20180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:34:59.187526 20180 hybrid_clock.cc:648] HybridClock initialized: now 1786250099187505 us; error 30 us; skew 500 ppm
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[19834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250099, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:34:59.190845 20180 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:34:59.191783 20180 webserver.cc:460] Webserver started at http://127.19.92.2:39683/ using document root <none> and password file <none>
I20260809 04:34:59.192344 20180 fs_manager.cc:359] Metadata directory not provided
I20260809 04:34:59.192414 20180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:34:59.192641 20180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:34:59.194864 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "73bf3edd396c4b59b07c562f091b5ccf"
format_stamp: "Formatted at 2026-08-09 04:34:59 on dist-test-slave-rmhg"
server_key: "dbf3225864000aa5d3675f42bbf3e124"
server_key_iv: "8e692be67f12fe765f8c56767066900b"
server_key_version: "encryptionkey@0"
I20260809 04:34:59.195417 20180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "73bf3edd396c4b59b07c562f091b5ccf"
format_stamp: "Formatted at 2026-08-09 04:34:59 on dist-test-slave-rmhg"
server_key: "dbf3225864000aa5d3675f42bbf3e124"
server_key_iv: "8e692be67f12fe765f8c56767066900b"
server_key_version: "encryptionkey@0"
I20260809 04:34:59.199249 20180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 04:34:59.201689 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:59.202927 20180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:34:59.203045 20180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "73bf3edd396c4b59b07c562f091b5ccf"
format_stamp: "Formatted at 2026-08-09 04:34:59 on dist-test-slave-rmhg"
server_key: "dbf3225864000aa5d3675f42bbf3e124"
server_key_iv: "8e692be67f12fe765f8c56767066900b"
server_key_version: "encryptionkey@0"
I20260809 04:34:59.203146 20180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:34:59.244468 20180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:34:59.248159 20180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:34:59.248351 20180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:34:59.248971 20180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:34:59.250030 20180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:34:59.250095 20180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:59.250160 20180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:34:59.250190 20180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:59.265259 20180 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:32893
I20260809 04:34:59.265447 20361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:32893 every 8 connection(s)
I20260809 04:34:59.266268 20180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:34:59.275355 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20180
I20260809 04:34:59.275460 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:34:59.275695 19824 external_mini_cluster.cc:1428] Setting key f1d908724e2a208ff94d756891d9cb0e
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[19834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250099, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:34:59.280793 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:59.268470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:55473 (local address 127.19.92.62:38901)
0809 04:34:59.268620 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 04:34:59.268624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:34:59.269851 (+ 1227us) server_negotiation.cc:407] Connection header received
0809 04:34:59.270876 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:34:59.270880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:34:59.270937 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:34:59.271012 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:34:59.272291 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.272825 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.274084 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.274305 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.274395 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:59.277067 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:34:59.277071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:34:59.277073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:34:59.277098 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:34:59.278524 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.279136 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.279139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.279141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.279242 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.279644 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.279647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.279649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.279872 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:34:59.279969 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:34:59.280493 (+ 524us) server_negotiation.cc:299] Negotiation successful
0809 04:34:59.280632 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:34:59.281795 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:59.268863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38901 (local address 127.19.92.2:55473)
0809 04:34:59.269705 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0809 04:34:59.269748 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:34:59.270672 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:34:59.271193 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:34:59.271201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:34:59.271617 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:34:59.272150 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:34:59.272165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.272944 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.272947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:34:59.273959 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:34:59.273969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.274445 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.274449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:34:59.274630 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:59.275191 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:34:59.275208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:34:59.276905 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:34:59.278642 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:34:59.278649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:34:59.278667 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:34:59.279010 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:34:59.279340 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:34:59.279344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:34:59.279346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:34:59.279558 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:34:59.279972 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:34:59.279978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:34:59.280235 (+ 257us) client_negotiation.cc:769] Sending connection context
0809 04:34:59.280460 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 04:34:59.280809 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":220,"threads_started":1}
I20260809 04:34:59.283072 20362 heartbeater.cc:347] Connected to a master server at 127.19.92.62:38901
I20260809 04:34:59.283357 20362 heartbeater.cc:464] Registering TS with master...
I20260809 04:34:59.283933 20362 heartbeater.cc:511] Master 127.19.92.62:38901 requested a full tablet report, sending...
I20260809 04:34:59.285187 19901 ts_manager.cc:194] Registered new tserver with Master: 73bf3edd396c4b59b07c562f091b5ccf (127.19.92.2:32893)
I20260809 04:34:59.285807 19901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:55473
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:34:59.306069 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38901
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:34:59.478789 20367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:34:59.479064 20367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:34:59.479112 20367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:34:59.483259 20367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:34:59.483323 20367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:34:59.483383 20367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:34:59.489356 20367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38901
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:34:59.490474 20367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:34:59.491593 20367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:34:59.503074 20367 server_base.cc:1034] running on GCE node
W20260809 04:34:59.503070 20377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:34:59.503311 20375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:34:59.503187 20373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:34:59.503816 20367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:34:59.504515 20367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:34:59.505684 20367 hybrid_clock.cc:648] HybridClock initialized: now 1786250099505666 us; error 53 us; skew 500 ppm
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[19834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250099, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:34:59.508728 20367 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:34:59.509614 20367 webserver.cc:460] Webserver started at http://127.19.92.3:40461/ using document root <none> and password file <none>
I20260809 04:34:59.510139 20367 fs_manager.cc:359] Metadata directory not provided
I20260809 04:34:59.510196 20367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:34:59.510372 20367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:34:59.512458 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "3e79e63ef1ff411fa4cc3b48d0006fc5"
format_stamp: "Formatted at 2026-08-09 04:34:59 on dist-test-slave-rmhg"
server_key: "562832a4090ebdff51d0610d4a626537"
server_key_iv: "7651d68f12d4ed965fbfd5cbdd967e92"
server_key_version: "encryptionkey@0"
I20260809 04:34:59.512988 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "3e79e63ef1ff411fa4cc3b48d0006fc5"
format_stamp: "Formatted at 2026-08-09 04:34:59 on dist-test-slave-rmhg"
server_key: "562832a4090ebdff51d0610d4a626537"
server_key_iv: "7651d68f12d4ed965fbfd5cbdd967e92"
server_key_version: "encryptionkey@0"
I20260809 04:34:59.517133 20367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 04:34:59.519634 20387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:59.520849 20367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:34:59.520959 20367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "3e79e63ef1ff411fa4cc3b48d0006fc5"
format_stamp: "Formatted at 2026-08-09 04:34:59 on dist-test-slave-rmhg"
server_key: "562832a4090ebdff51d0610d4a626537"
server_key_iv: "7651d68f12d4ed965fbfd5cbdd967e92"
server_key_version: "encryptionkey@0"
I20260809 04:34:59.521054 20367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:34:59.537089 20367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:34:59.540457 20367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:34:59.540704 20367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:34:59.541548 20367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:34:59.542557 20367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:34:59.542605 20367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:59.542647 20367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:34:59.542675 20367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:34:59.562769 20367 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:35853
I20260809 04:34:59.562875 20557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:35853 every 8 connection(s)
I20260809 04:34:59.563936 20367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:34:59.565063 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20367
I20260809 04:34:59.565192 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:34:59.565430 19824 external_mini_cluster.cc:1428] Setting key 7c02188e232497d57bfa4b2760484f1d
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[19834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250099, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:34:59.580214 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:59.567311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:55531 (local address 127.19.92.62:38901)
0809 04:34:59.567433 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:34:59.567436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:34:59.569060 (+ 1624us) server_negotiation.cc:407] Connection header received
0809 04:34:59.570217 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:34:59.570221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:34:59.570278 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:34:59.570350 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:34:59.571866 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.572397 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.573728 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.573943 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.574030 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:59.576841 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:34:59.576844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:34:59.576846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:34:59.576873 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:34:59.578083 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.578554 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.578557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.578558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.578638 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.579136 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.579139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.579141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.579353 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:34:59.579428 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 04:34:59.579962 (+ 534us) server_negotiation.cc:299] Negotiation successful
0809 04:34:59.580084 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 04:34:59.581233 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:59.567998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38901 (local address 127.19.92.3:55531)
0809 04:34:59.568906 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0809 04:34:59.568954 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 04:34:59.570009 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:34:59.570538 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:34:59.570547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:34:59.571080 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:34:59.571717 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:34:59.571734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.572533 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.572536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:34:59.573607 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:34:59.573618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.574089 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.574093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:34:59.574307 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:59.574962 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:34:59.574979 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:34:59.576675 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:34:59.578185 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:34:59.578190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:34:59.578201 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:34:59.578452 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:34:59.578884 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:34:59.578887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:34:59.578888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:34:59.579052 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:34:59.579451 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:34:59.579458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:34:59.579721 (+ 263us) client_negotiation.cc:769] Sending connection context
0809 04:34:59.579934 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 04:34:59.580294 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":161,"threads_started":1}
I20260809 04:34:59.582316 20558 heartbeater.cc:347] Connected to a master server at 127.19.92.62:38901
I20260809 04:34:59.582538 20558 heartbeater.cc:464] Registering TS with master...
I20260809 04:34:59.583029 20558 heartbeater.cc:511] Master 127.19.92.62:38901 requested a full tablet report, sending...
I20260809 04:34:59.583999 19901 ts_manager.cc:194] Registered new tserver with Master: 3e79e63ef1ff411fa4cc3b48d0006fc5 (127.19.92.3:35853)
I20260809 04:34:59.584582 19901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:55531
I20260809 04:34:59.592018 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[19834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250099, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[19834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250099, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:34:59.622140 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:59.612544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.19.92.1:37441)
0809 04:34:59.612815 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 04:34:59.612823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:34:59.612848 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:34:59.612941 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:34:59.612946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:34:59.613144 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:34:59.613306 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:34:59.613969 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.614841 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.615678 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.616125 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.616207 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:59.618655 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:34:59.618667 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:34:59.618682 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:34:59.618708 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:34:59.620401 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.620789 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.620794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.620795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.620877 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.621101 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.621104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.621105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.621430 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:34:59.621619 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 04:34:59.621765 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 04:34:59.621910 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
W20260809 04:34:59.623167 20046 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60356
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[19834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[19834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:34:59.651789 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:34:59.643355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.19.92.1:37441)
0809 04:34:59.643505 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 04:34:59.643509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:34:59.643615 (+ 106us) server_negotiation.cc:407] Connection header received
0809 04:34:59.643777 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:34:59.643780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:34:59.643823 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:34:59.643889 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:34:59.644662 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.645226 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.646102 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:34:59.646343 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:34:59.646450 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:34:59.648864 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:34:59.648867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:34:59.648869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:34:59.648899 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:34:59.650206 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.650625 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.650629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.650631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.650711 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:34:59.650959 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:34:59.650962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:34:59.650964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:34:59.651149 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:34:59.651271 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 04:34:59.651508 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 04:34:59.651634 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:34:59.652832 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 19981
I20260809 04:34:59.666868 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20180
I20260809 04:34:59.676937 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20367
I20260809 04:34:59.689630 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 19852
2026-08-09T04:34:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2641 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[20579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[20579](info): set up 2 sockets
Aug 09 04:34:59 dist-test-slave-rmhg krb5kdc[20579](info): commencing operation
krb5kdc: starting...
W20260809 04:35:00.818593 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:00 dist-test-slave-rmhg krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:00.946326 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:33893
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:33893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:01.059269 20595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:01.059559 20595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:01.059617 20595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:01.063706 20595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:01.063772 20595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:01.063798 20595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:01.063818 20595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:01.063840 20595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:01.069291 20595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:33893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:01.070405 20595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:01.071444 20595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:01.079759 20604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:01.079756 20603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:01.079756 20606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:01.079919 20595 server_base.cc:1034] running on GCE node
I20260809 04:35:01.080610 20595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:01.081773 20595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:01.082953 20595 hybrid_clock.cc:648] HybridClock initialized: now 1786250101082933 us; error 31 us; skew 500 ppm
Aug 09 04:35:01 dist-test-slave-rmhg krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250101, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:01.085831 20595 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:01.086732 20595 webserver.cc:460] Webserver started at http://127.19.92.62:40977/ using document root <none> and password file <none>
I20260809 04:35:01.087330 20595 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:01.087401 20595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:01.087615 20595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:01.089890 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "1862cd51aa7e45e9a449e28e65defc20"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "503d9e78eb7a608a5faa46cc2383b847"
server_key_iv: "468ef1c4723dd73c6e9022b526d14144"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.090482 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "1862cd51aa7e45e9a449e28e65defc20"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "503d9e78eb7a608a5faa46cc2383b847"
server_key_iv: "468ef1c4723dd73c6e9022b526d14144"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.094370 20595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 04:35:01.096939 20614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.098230 20595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:35:01.098351 20595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "1862cd51aa7e45e9a449e28e65defc20"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "503d9e78eb7a608a5faa46cc2383b847"
server_key_iv: "468ef1c4723dd73c6e9022b526d14144"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.098455 20595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:01.113471 20595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:01.116480 20595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:01.116668 20595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:01.123711 20595 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:33893
I20260809 04:35:01.124089 20691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:33893 every 8 connection(s)
I20260809 04:35:01.124724 20595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:01.127684 20693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:01.133563 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20595
I20260809 04:35:01.133569 20693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20: Bootstrap starting.
I20260809 04:35:01.133687 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:01.133970 19824 external_mini_cluster.cc:1428] Setting key 7a17b452c1504aa075806ce609a9926d
I20260809 04:35:01.136315 20693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:01.137058 20693 log.cc:824] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:01.138926 20693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20: No bootstrap required, opened a new log
I20260809 04:35:01.142553 20693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1862cd51aa7e45e9a449e28e65defc20" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33893 } }
I20260809 04:35:01.142843 20693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 09 04:35:01 dist-test-slave-rmhg krb5kdc[20579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:01.142901 20693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1862cd51aa7e45e9a449e28e65defc20, State: Initialized, Role: FOLLOWER
I20260809 04:35:01.143321 20693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1862cd51aa7e45e9a449e28e65defc20" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33893 } }
I20260809 04:35:01.143453 20693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:01.143520 20693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:01.143611 20693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:01.144656 20693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1862cd51aa7e45e9a449e28e65defc20" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33893 } }
I20260809 04:35:01.144990 20693 leader_election.cc:302] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1862cd51aa7e45e9a449e28e65defc20; no voters:
I20260809 04:35:01.145309 20693 leader_election.cc:288] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:01.145404 20698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:01.145654 20698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [term 1 LEADER]: Becoming Leader. State: Replica: 1862cd51aa7e45e9a449e28e65defc20, State: Running, Role: LEADER
I20260809 04:35:01.145989 20698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1862cd51aa7e45e9a449e28e65defc20" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33893 } }
I20260809 04:35:01.146581 20693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:01.148389 20700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1862cd51aa7e45e9a449e28e65defc20. Latest consensus state: current_term: 1 leader_uuid: "1862cd51aa7e45e9a449e28e65defc20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1862cd51aa7e45e9a449e28e65defc20" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33893 } } }
I20260809 04:35:01.148494 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1862cd51aa7e45e9a449e28e65defc20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1862cd51aa7e45e9a449e28e65defc20" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33893 } } }
I20260809 04:35:01.148595 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:01.148540 20700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:01.149031 20714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:01.149305 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.135538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.19.92.62:33893)
0809 04:35:01.136019 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:01.136034 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:01.136072 (+ 38us) server_negotiation.cc:407] Connection header received
0809 04:35:01.136758 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:01.136778 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:01.137139 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:01.137510 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:01.138297 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.139328 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.140186 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.140665 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.140812 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.143469 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:01.143476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:01.143494 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:01.143528 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:01.145320 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.145855 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.145861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.145867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.146025 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.146407 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.146410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.146412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.146930 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:01.147148 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:01.147448 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 04:35:01.147768 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":147,"threads_started":1}
I20260809 04:35:01.152204 20714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:01.158237 20714 catalog_manager.cc:1347] Generated new cluster ID: 0f448b0ebf6f4405b9ecc9645359d9cf
I20260809 04:35:01.158299 20714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:01.168818 20714 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:01.169880 20714 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:01.179546 20714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1862cd51aa7e45e9a449e28e65defc20: Generated new TSK 0
I20260809 04:35:01.180300 20714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:01.212002 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:33893
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:01.330566 20725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:01.330868 20725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:01.330915 20725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:01.335047 20725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:01.335122 20725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:01.335186 20725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:01.340775 20725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:33893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:01.342000 20725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:01.343029 20725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:01.352977 20737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:01.352979 20734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:01.353176 20725 server_base.cc:1034] running on GCE node
W20260809 04:35:01.353332 20735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:01.353735 20725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:01.354365 20725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:01.355530 20725 hybrid_clock.cc:648] HybridClock initialized: now 1786250101355510 us; error 36 us; skew 500 ppm
Aug 09 04:35:01 dist-test-slave-rmhg krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250101, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:01.361385 20725 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:01.362347 20725 webserver.cc:460] Webserver started at http://127.19.92.1:37961/ using document root <none> and password file <none>
I20260809 04:35:01.362919 20725 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:01.362988 20725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:01.363214 20725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:01.365494 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "fd469cb81e2843c1b74008a424a3cb59"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "bb78bf663e32d466c997f08363a3ed5c"
server_key_iv: "30d76c95390ed7cf8eeee338f838d30e"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.366086 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "fd469cb81e2843c1b74008a424a3cb59"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "bb78bf663e32d466c997f08363a3ed5c"
server_key_iv: "30d76c95390ed7cf8eeee338f838d30e"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.370151 20725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:35:01.372830 20745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.374181 20725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:01.374295 20725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "fd469cb81e2843c1b74008a424a3cb59"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "bb78bf663e32d466c997f08363a3ed5c"
server_key_iv: "30d76c95390ed7cf8eeee338f838d30e"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.374394 20725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:01.391234 20725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:01.394312 20725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:01.394505 20725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:01.395110 20725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:01.395964 20725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:01.396013 20725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.396067 20725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:01.396097 20725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.405025 20725 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:39141
I20260809 04:35:01.405102 20901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:39141 every 8 connection(s)
I20260809 04:35:01.406006 20725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:01.409716 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20725
I20260809 04:35:01.409821 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:01.410071 19824 external_mini_cluster.cc:1428] Setting key 9152954c1418fe4ce3bddaa94989c776
Aug 09 04:35:01 dist-test-slave-rmhg krb5kdc[20579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250101, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:01.419608 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.407917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:42577 (local address 127.19.92.62:33893)
0809 04:35:01.408097 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:01.408101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:01.409026 (+ 925us) server_negotiation.cc:407] Connection header received
0809 04:35:01.409948 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:01.409952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:01.410018 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:01.410096 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:01.411441 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.411969 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.413100 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.413470 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.413573 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.416069 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:01.416073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:01.416075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:01.416101 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:01.417528 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.418016 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.418019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.418020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.418099 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.418433 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.418436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.418437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.418624 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:01.418712 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:01.419240 (+ 528us) server_negotiation.cc:299] Negotiation successful
0809 04:35:01.419378 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:35:01.420485 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.408335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:33893 (local address 127.19.92.1:42577)
0809 04:35:01.408837 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:01.408881 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:01.409735 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:01.410289 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:01.410296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:01.410744 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:01.411265 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.411285 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.412095 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.412097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.412986 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.412997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.413631 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.413634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.413794 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.414370 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:01.414387 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:01.415895 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:01.417636 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:01.417642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:01.417659 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:01.417911 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:01.418182 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:01.418185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:01.418187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:01.418336 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:01.418727 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:01.418734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:01.418999 (+ 265us) client_negotiation.cc:769] Sending connection context
0809 04:35:01.419174 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 04:35:01.419538 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260809 04:35:01.421751 20902 heartbeater.cc:347] Connected to a master server at 127.19.92.62:33893
I20260809 04:35:01.422006 20902 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:01.422513 20902 heartbeater.cc:511] Master 127.19.92.62:33893 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:35:01.424182 20638 ts_manager.cc:194] Registered new tserver with Master: fd469cb81e2843c1b74008a424a3cb59 (127.19.92.1:39141)
I20260809 04:35:01.425458 20638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:42577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:01.439615 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:33893
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:01.554441 20911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:01.554718 20911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:01.554762 20911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:01.558754 20911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:01.558821 20911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:01.558884 20911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:01.564230 20911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:33893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:01.565441 20911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:01.566507 20911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:01.577064 20918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:01.577065 20917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:01.577064 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:01.577380 20911 server_base.cc:1034] running on GCE node
I20260809 04:35:01.577889 20911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:01.578631 20911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:01.579820 20911 hybrid_clock.cc:648] HybridClock initialized: now 1786250101579781 us; error 58 us; skew 500 ppm
Aug 09 04:35:01 dist-test-slave-rmhg krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250101, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:01.582724 20911 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:01.583631 20911 webserver.cc:460] Webserver started at http://127.19.92.2:33393/ using document root <none> and password file <none>
I20260809 04:35:01.584203 20911 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:01.584270 20911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:01.584503 20911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:01.586728 20911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "92470317f15a4b4882f36adf350b1171"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "117c57600b329214190626d03de95d1b"
server_key_iv: "38b6eca05d547127a8c041c106bcdf97"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.587273 20911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "92470317f15a4b4882f36adf350b1171"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "117c57600b329214190626d03de95d1b"
server_key_iv: "38b6eca05d547127a8c041c106bcdf97"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.591229 20911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:35:01.593861 20931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.595150 20911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:01.595263 20911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "92470317f15a4b4882f36adf350b1171"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "117c57600b329214190626d03de95d1b"
server_key_iv: "38b6eca05d547127a8c041c106bcdf97"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.595362 20911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:01.605581 20911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:01.609014 20911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:01.609246 20911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:01.609884 20911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:01.610787 20911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:01.610837 20911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.610893 20911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:01.610921 20911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.621274 20911 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:33883
I20260809 04:35:01.621299 21094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:33883 every 8 connection(s)
I20260809 04:35:01.622217 20911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:01.627126 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20911
I20260809 04:35:01.627249 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:01.627592 19824 external_mini_cluster.cc:1428] Setting key 3b567d4a2118b83e332c0cfa17c37731
Aug 09 04:35:01 dist-test-slave-rmhg krb5kdc[20579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250101, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:01.636315 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.624069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:41603 (local address 127.19.92.62:33893)
0809 04:35:01.624194 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:01.624197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:01.625064 (+ 867us) server_negotiation.cc:407] Connection header received
0809 04:35:01.626029 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:01.626033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:01.626092 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:01.626167 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:01.627598 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.628248 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.629752 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.629973 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.630059 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.632749 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:01.632752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:01.632754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:01.632778 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:01.634126 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.634722 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.634725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.634726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.634802 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.635157 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.635159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.635160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.635339 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:01.635411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:01.636082 (+ 671us) server_negotiation.cc:299] Negotiation successful
0809 04:35:01.636186 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:35:01.637558 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.624431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:33893 (local address 127.19.92.2:41603)
0809 04:35:01.624901 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:01.624945 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:01.625814 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:01.626362 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:01.626370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:01.626730 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:01.627425 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.627447 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.628381 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.628384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.629637 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.629648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.630123 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.630127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.630324 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.630929 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:01.630949 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:01.632574 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:01.634236 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:01.634242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:01.634260 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:01.634629 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:01.634877 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:01.634880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:01.634882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:01.635069 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:01.635434 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:01.635441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:01.635807 (+ 366us) client_negotiation.cc:769] Sending connection context
0809 04:35:01.636043 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 04:35:01.636413 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:35:01.638803 21096 heartbeater.cc:347] Connected to a master server at 127.19.92.62:33893
I20260809 04:35:01.639076 21096 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:01.639608 21096 heartbeater.cc:511] Master 127.19.92.62:33893 requested a full tablet report, sending...
I20260809 04:35:01.640698 20638 ts_manager.cc:194] Registered new tserver with Master: 92470317f15a4b4882f36adf350b1171 (127.19.92.2:33883)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:35:01.641378 20638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:41603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:01.657730 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:33893
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:01.776530 21102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:01.776809 21102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:01.776856 21102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:01.781013 21102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:01.781090 21102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:01.781176 21102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:01.786767 21102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:33893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:01.787899 21102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:01.788956 21102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:01.801582 21111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:01.801648 21108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:01.801626 21113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:01.801930 21102 server_base.cc:1034] running on GCE node
I20260809 04:35:01.802341 21102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:01.803009 21102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:01.804171 21102 hybrid_clock.cc:648] HybridClock initialized: now 1786250101804141 us; error 44 us; skew 500 ppm
Aug 09 04:35:01 dist-test-slave-rmhg krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250101, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:01.807003 21102 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:01.808007 21102 webserver.cc:460] Webserver started at http://127.19.92.3:39785/ using document root <none> and password file <none>
I20260809 04:35:01.808754 21102 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:01.808842 21102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:01.809074 21102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:01.811301 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "51552eb64a8a4e86aeb1ec6b6df3d295"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "8037245df5d8d70e229e88639931c4c1"
server_key_iv: "057769a88a1c9d9af614530d448ceb1d"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.811893 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "51552eb64a8a4e86aeb1ec6b6df3d295"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "8037245df5d8d70e229e88639931c4c1"
server_key_iv: "057769a88a1c9d9af614530d448ceb1d"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.815838 21102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 04:35:01.818544 21122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.819808 21102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:35:01.819931 21102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "51552eb64a8a4e86aeb1ec6b6df3d295"
format_stamp: "Formatted at 2026-08-09 04:35:01 on dist-test-slave-rmhg"
server_key: "8037245df5d8d70e229e88639931c4c1"
server_key_iv: "057769a88a1c9d9af614530d448ceb1d"
server_key_version: "encryptionkey@0"
I20260809 04:35:01.820029 21102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:01.832047 21102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:01.835053 21102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:01.835229 21102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:01.835810 21102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:01.836650 21102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:01.836699 21102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.836752 21102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:01.836782 21102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:01.857030 21102 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:37355
I20260809 04:35:01.857051 21281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:37355 every 8 connection(s)
I20260809 04:35:01.858096 21102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:01.865680 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21102
I20260809 04:35:01.865821 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:01.866101 19824 external_mini_cluster.cc:1428] Setting key aa1d0e77dff2fd2408b4a249b31beeeb
Aug 09 04:35:01 dist-test-slave-rmhg krb5kdc[20579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250101, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:01.873749 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.860217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:52907 (local address 127.19.92.62:33893)
0809 04:35:01.860383 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:01.860388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:01.862215 (+ 1827us) server_negotiation.cc:407] Connection header received
0809 04:35:01.863337 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:01.863340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:01.863390 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:01.863470 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:01.865125 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.865714 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.867343 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.867640 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.867725 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.870456 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:01.870460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:01.870462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:01.870488 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:01.871773 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.872264 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.872269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.872270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.872372 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.872668 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.872670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.872672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.872855 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:01.872938 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:01.873517 (+ 579us) server_negotiation.cc:299] Negotiation successful
0809 04:35:01.873623 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:35:01.874720 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.860627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:33893 (local address 127.19.92.3:52907)
0809 04:35:01.861981 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:01.862031 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:01.863125 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:01.863636 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:01.863643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:01.864042 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:01.864970 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.864989 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.865846 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.865848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.867232 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.867243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.867791 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.867795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.868052 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.868724 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:01.868741 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:01.870290 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:01.871900 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:01.871905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:01.871918 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:01.872171 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:01.872453 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:01.872455 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:01.872457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:01.872597 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:01.872971 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:01.872976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:01.873284 (+ 308us) client_negotiation.cc:769] Sending connection context
0809 04:35:01.873456 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 04:35:01.873734 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":212,"threads_started":1}
I20260809 04:35:01.876070 21282 heartbeater.cc:347] Connected to a master server at 127.19.92.62:33893
I20260809 04:35:01.876294 21282 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:01.876787 21282 heartbeater.cc:511] Master 127.19.92.62:33893 requested a full tablet report, sending...
I20260809 04:35:01.877853 20638 ts_manager.cc:194] Registered new tserver with Master: 51552eb64a8a4e86aeb1ec6b6df3d295 (127.19.92.3:37355)
I20260809 04:35:01.878424 20638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:52907
I20260809 04:35:01.880050 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:35:01.891338 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.883911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.19.92.62:33893)
0809 04:35:01.884078 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:01.884082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:01.884424 (+ 342us) server_negotiation.cc:407] Connection header received
0809 04:35:01.884535 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:01.884537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:01.884577 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:01.884640 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:01.885364 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.885867 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.886735 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.886941 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.887022 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.888422 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:01.888426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:01.888428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:01.888452 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:01.889806 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.890240 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.890243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.890244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.890331 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:01.890581 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:01.890583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:01.890584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:01.890802 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:01.890903 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:01.891078 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 04:35:01.891180 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:35:01.896453 20638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:01.899760 20638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:01.915544 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.908465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33883 (local address 127.0.0.1:34472)
0809 04:35:01.908813 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:01.908863 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:01.909175 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:01.910607 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:01.910611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:01.910630 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:01.910791 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.910799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.912937 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.912940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.913808 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.913820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.915044 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.915047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.915122 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.915141 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:35:01.915182 (+ 41us) client_negotiation.cc:240] Negotiation successful
0809 04:35:01.915356 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":57,"thread_start_us":80,"threads_started":1}
I20260809 04:35:01.915544 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.908677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:39141 (local address 127.0.0.1:54052)
0809 04:35:01.909329 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:01.909342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:01.909476 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:01.910303 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:01.910310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:01.910338 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:01.910505 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.910515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.912090 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.912094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.913602 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.913613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.914909 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.914913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.915028 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.915076 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 04:35:01.915184 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 04:35:01.915350 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":28,"thread_start_us":233,"threads_started":1}
I20260809 04:35:01.916018 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.910009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:37355 (local address 127.0.0.1:41004)
0809 04:35:01.910412 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:01.910425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:01.910553 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:01.911069 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:01.911072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:01.911090 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:01.911172 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.911179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.912791 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.912794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.913869 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:01.913877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.915213 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.915217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:01.915305 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.915328 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:35:01.915395 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 04:35:01.915480 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":167,"threads_started":1}
I20260809 04:35:01.916258 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.909329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.19.92.2:33883)
0809 04:35:01.910136 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:01.910151 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:01.910176 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:35:01.910254 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:01.910260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:01.910453 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:01.910951 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:01.911207 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.912711 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.913941 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.914889 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.915749 (+ 860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.915895 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:01.915947 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 04:35:01.916051 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":434,"threads_started":1}
I20260809 04:35:01.916581 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.908845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.19.92.1:39141)
0809 04:35:01.909717 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:01.909774 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:01.909801 (+ 27us) server_negotiation.cc:407] Connection header received
0809 04:35:01.909884 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:01.909889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:01.910112 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:01.910665 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:01.910860 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.911879 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.913734 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.914771 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.914892 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.915059 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:01.916246 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0809 04:35:01.916348 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":144,"threads_started":1}
I20260809 04:35:01.917378 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:01.909644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.19.92.3:37355)
0809 04:35:01.910227 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:01.910239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:01.910571 (+ 332us) server_negotiation.cc:407] Connection header received
0809 04:35:01.910659 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:01.910665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:01.910876 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:01.911082 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:01.911305 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.912596 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.914036 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:01.915093 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:01.915197 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:01.915344 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:01.915427 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 04:35:01.915523 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":156,"threads_started":1}
I20260809 04:35:01.919549 20808 tablet_service.cc:1467] Processing CreateTablet for tablet f79e9e84abc5402d9c64f45c205d98c8 (DEFAULT_TABLE table=test-table [id=8981cbc359f8493aaa21420778283977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:01.919842 21004 tablet_service.cc:1467] Processing CreateTablet for tablet f79e9e84abc5402d9c64f45c205d98c8 (DEFAULT_TABLE table=test-table [id=8981cbc359f8493aaa21420778283977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:01.920274 21189 tablet_service.cc:1467] Processing CreateTablet for tablet f79e9e84abc5402d9c64f45c205d98c8 (DEFAULT_TABLE table=test-table [id=8981cbc359f8493aaa21420778283977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:01.920888 21004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79e9e84abc5402d9c64f45c205d98c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:01.920933 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79e9e84abc5402d9c64f45c205d98c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:01.921401 21189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79e9e84abc5402d9c64f45c205d98c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:01.929294 21309 tablet_bootstrap.cc:492] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171: Bootstrap starting.
I20260809 04:35:01.930914 21311 tablet_bootstrap.cc:492] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59: Bootstrap starting.
I20260809 04:35:01.932848 21309 tablet_bootstrap.cc:654] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:01.933564 21313 tablet_bootstrap.cc:492] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295: Bootstrap starting.
I20260809 04:35:01.934180 21311 tablet_bootstrap.cc:654] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:01.934623 21309 log.cc:824] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:01.935392 21311 log.cc:824] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:01.936679 21313 tablet_bootstrap.cc:654] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:01.936853 21309 tablet_bootstrap.cc:492] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171: No bootstrap required, opened a new log
I20260809 04:35:01.937045 21309 ts_tablet_manager.cc:1397] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 04:35:01.937798 21311 tablet_bootstrap.cc:492] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59: No bootstrap required, opened a new log
I20260809 04:35:01.938061 21311 ts_tablet_manager.cc:1397] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 04:35:01.938827 21313 log.cc:824] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:01.941221 21313 tablet_bootstrap.cc:492] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295: No bootstrap required, opened a new log
I20260809 04:35:01.941416 21313 ts_tablet_manager.cc:1397] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 04:35:01.941690 21309 raft_consensus.cc:348] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:01.941967 21309 raft_consensus.cc:374] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:01.942034 21309 raft_consensus.cc:729] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92470317f15a4b4882f36adf350b1171, State: Initialized, Role: FOLLOWER
I20260809 04:35:01.942456 21309 consensus_queue.cc:260] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:01.943212 21096 heartbeater.cc:503] Master 127.19.92.62:33893 was elected leader, sending a full tablet report...
I20260809 04:35:01.943441 21309 ts_tablet_manager.cc:1428] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 04:35:01.943974 21311 raft_consensus.cc:348] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:01.944234 21311 raft_consensus.cc:374] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:01.944283 21311 raft_consensus.cc:729] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd469cb81e2843c1b74008a424a3cb59, State: Initialized, Role: FOLLOWER
I20260809 04:35:01.944759 21311 consensus_queue.cc:260] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:01.945518 20902 heartbeater.cc:503] Master 127.19.92.62:33893 was elected leader, sending a full tablet report...
I20260809 04:35:01.946587 21313 raft_consensus.cc:348] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:01.946930 21311 ts_tablet_manager.cc:1428] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260809 04:35:01.947031 21313 raft_consensus.cc:374] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:01.947091 21313 raft_consensus.cc:729] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51552eb64a8a4e86aeb1ec6b6df3d295, State: Initialized, Role: FOLLOWER
I20260809 04:35:01.947557 21313 consensus_queue.cc:260] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:01.948544 21282 heartbeater.cc:503] Master 127.19.92.62:33893 was elected leader, sending a full tablet report...
I20260809 04:35:01.948747 21313 ts_tablet_manager.cc:1428] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260809 04:35:02.109477 21283 tablet.cc:2369] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:35:02.123374 21098 tablet.cc:2369] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:35:02.153214 21318 raft_consensus.cc:482] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:02.153455 21318 raft_consensus.cc:504] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:02.154690 21318 leader_election.cc:288] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92470317f15a4b4882f36adf350b1171 (127.19.92.2:33883), 51552eb64a8a4e86aeb1ec6b6df3d295 (127.19.92.3:37355)
W20260809 04:35:02.157302 20903 tablet.cc:2369] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:35:02.158747 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:02.154941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33883 (local address 127.19.92.1:47187)
0809 04:35:02.155265 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:02.155277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:02.155442 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:02.155756 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:02.155758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:02.155781 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:02.155922 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:02.155930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.156632 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.156635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:02.157640 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:02.157650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.158322 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.158325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:02.158426 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:02.158453 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 04:35:02.158527 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:35:02.158593 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260809 04:35:02.158757 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:02.155023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:47187 (local address 127.19.92.2:33883)
0809 04:35:02.155161 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:02.155166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:02.155407 (+ 241us) server_negotiation.cc:407] Connection header received
0809 04:35:02.155581 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:02.155585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:02.155641 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:02.155728 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:02.156042 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.156533 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.157771 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.158202 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.158285 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:02.158338 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:02.158579 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 04:35:02.158654 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:35:02.159463 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79e9e84abc5402d9c64f45c205d98c8" candidate_uuid: "fd469cb81e2843c1b74008a424a3cb59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92470317f15a4b4882f36adf350b1171" is_pre_election: true
I20260809 04:35:02.159773 21030 raft_consensus.cc:2454] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd469cb81e2843c1b74008a424a3cb59 in term 0.
I20260809 04:35:02.160008 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:02.155341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:37355 (local address 127.19.92.1:53751)
0809 04:35:02.155895 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:02.155908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:02.156148 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:02.156370 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:02.156373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:02.156389 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:02.156642 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:02.156650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.157542 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.157545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:02.158924 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:02.158933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.159617 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.159621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:02.159732 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:02.159753 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:35:02.159816 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 04:35:02.159878 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":270,"threads_started":1}
I20260809 04:35:02.160035 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:02.155533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:53751 (local address 127.19.92.3:37355)
0809 04:35:02.155656 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:02.155659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:02.156129 (+ 470us) server_negotiation.cc:407] Connection header received
0809 04:35:02.156247 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:02.156249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:02.156292 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:02.156352 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:02.156762 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.157378 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.159047 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.159499 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.159593 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:02.159652 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:02.159845 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 04:35:02.159918 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:35:02.160290 20748 leader_election.cc:302] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92470317f15a4b4882f36adf350b1171, fd469cb81e2843c1b74008a424a3cb59; no voters:
I20260809 04:35:02.160560 21318 raft_consensus.cc:2781] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:02.160660 21318 raft_consensus.cc:482] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:02.160717 21318 raft_consensus.cc:3037] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:02.161026 21214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79e9e84abc5402d9c64f45c205d98c8" candidate_uuid: "fd469cb81e2843c1b74008a424a3cb59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" is_pre_election: true
I20260809 04:35:02.161305 21214 raft_consensus.cc:2454] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd469cb81e2843c1b74008a424a3cb59 in term 0.
I20260809 04:35:02.161934 21318 raft_consensus.cc:504] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:02.162248 21318 leader_election.cc:288] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [CANDIDATE]: Term 1 election: Requested vote from peers 92470317f15a4b4882f36adf350b1171 (127.19.92.2:33883), 51552eb64a8a4e86aeb1ec6b6df3d295 (127.19.92.3:37355)
I20260809 04:35:02.162595 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79e9e84abc5402d9c64f45c205d98c8" candidate_uuid: "fd469cb81e2843c1b74008a424a3cb59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92470317f15a4b4882f36adf350b1171"
I20260809 04:35:02.162631 21214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79e9e84abc5402d9c64f45c205d98c8" candidate_uuid: "fd469cb81e2843c1b74008a424a3cb59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295"
I20260809 04:35:02.162730 21214 raft_consensus.cc:3037] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:02.162730 21030 raft_consensus.cc:3037] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:02.164045 21214 raft_consensus.cc:2454] T f79e9e84abc5402d9c64f45c205d98c8 P 51552eb64a8a4e86aeb1ec6b6df3d295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd469cb81e2843c1b74008a424a3cb59 in term 1.
I20260809 04:35:02.164045 21030 raft_consensus.cc:2454] T f79e9e84abc5402d9c64f45c205d98c8 P 92470317f15a4b4882f36adf350b1171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd469cb81e2843c1b74008a424a3cb59 in term 1.
I20260809 04:35:02.164331 20748 leader_election.cc:302] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51552eb64a8a4e86aeb1ec6b6df3d295, fd469cb81e2843c1b74008a424a3cb59; no voters:
I20260809 04:35:02.164505 21318 raft_consensus.cc:2781] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:02.164733 21318 raft_consensus.cc:686] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [term 1 LEADER]: Becoming Leader. State: Replica: fd469cb81e2843c1b74008a424a3cb59, State: Running, Role: LEADER
I20260809 04:35:02.165064 21318 consensus_queue.cc:237] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } }
I20260809 04:35:02.168349 20638 catalog_manager.cc:5495] T f79e9e84abc5402d9c64f45c205d98c8 P fd469cb81e2843c1b74008a424a3cb59 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd469cb81e2843c1b74008a424a3cb59 (127.19.92.1). New cstate: current_term: 1 leader_uuid: "fd469cb81e2843c1b74008a424a3cb59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92470317f15a4b4882f36adf350b1171" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51552eb64a8a4e86aeb1ec6b6df3d295" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd469cb81e2843c1b74008a424a3cb59" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39141 } health_report { overall_health: HEALTHY } } }
Aug 09 04:35:02 dist-test-slave-rmhg krb5kdc[20579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:35:02.213115 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:02.204052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.19.92.1:39141)
0809 04:35:02.204224 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:02.204227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:02.204451 (+ 224us) server_negotiation.cc:407] Connection header received
0809 04:35:02.204567 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:02.204570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:02.204618 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:02.204688 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:02.205412 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.205967 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.206773 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.207007 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.207097 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:02.209681 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:02.209696 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:02.209715 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:02.209744 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:02.211312 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:02.211754 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:02.211761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:02.211762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:02.211850 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:02.212122 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:02.212125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:02.212127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:02.212403 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:02.212650 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:02.212802 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 04:35:02.212929 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 04:35:02 dist-test-slave-rmhg krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:35:02 dist-test-slave-rmhg krb5kdc[20579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:35:02.242095 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:02.233277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.19.92.1:39141)
0809 04:35:02.233449 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:02.233454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:02.233811 (+ 357us) server_negotiation.cc:407] Connection header received
0809 04:35:02.233978 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:02.233981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:02.234028 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:02.234101 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:02.234817 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.235347 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.236227 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.236473 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.236567 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:02.238976 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:02.238980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:02.238982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:02.239010 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:02.240332 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:02.240793 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:02.240796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:02.240797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:02.240881 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:02.241143 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:02.241146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:02.241179 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:02.241365 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:02.241517 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:02.241801 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 04:35:02.241934 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 04:35:02 dist-test-slave-rmhg krb5kdc[20579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:35:02 dist-test-slave-rmhg krb5kdc[20579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:35:02.270053 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:02.261941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.19.92.1:39141)
0809 04:35:02.262126 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:02.262131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:02.262401 (+ 270us) server_negotiation.cc:407] Connection header received
0809 04:35:02.262592 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:02.262596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:02.262651 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:02.262739 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:02.263545 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.264071 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.264952 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:02.265214 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:02.265297 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:02.267623 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:02.267627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:02.267629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:02.267653 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:02.268713 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:02.269066 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:02.269069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:02.269070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:02.269134 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:02.269390 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:02.269393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:02.269394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:02.269580 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:02.269696 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:02.269836 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 04:35:02.269929 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260809 04:35:02.271404 20780 tablet_service.cc:2908] Rejecting scan request for tablet f79e9e84abc5402d9c64f45c205d98c8: Uninitialized: clean time has not yet been initialized
I20260809 04:35:02.273485 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20725
I20260809 04:35:02.284792 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20911
I20260809 04:35:02.297788 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21102
I20260809 04:35:02.309732 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20595
2026-08-09T04:35:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2623 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:02 dist-test-slave-rmhg krb5kdc[21346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:02 dist-test-slave-rmhg krb5kdc[21346](info): set up 2 sockets
Aug 09 04:35:02 dist-test-slave-rmhg krb5kdc[21346](info): commencing operation
krb5kdc: starting...
W20260809 04:35:03.453382 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:03 dist-test-slave-rmhg krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:03.581305 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:45231
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:45231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:03.694162 21366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:03.694429 21366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:03.694473 21366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:03.698585 21366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:03.698638 21366 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 04:35:03.698654 21366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:03.698671 21366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:03.698688 21366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:03.698716 21366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:03.704010 21366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:45231
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:45231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:03.705085 21366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:03.706112 21366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:03.714385 21376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:03.714398 21375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:03.714495 21366 server_base.cc:1034] running on GCE node
W20260809 04:35:03.714393 21379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:03.715137 21366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:03.716233 21366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:03.717389 21366 hybrid_clock.cc:648] HybridClock initialized: now 1786250103717368 us; error 33 us; skew 500 ppm
Aug 09 04:35:03 dist-test-slave-rmhg krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250103, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:03.720050 21366 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:03.720924 21366 webserver.cc:460] Webserver started at http://127.19.92.62:42817/ using document root <none> and password file <none>
I20260809 04:35:03.721494 21366 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:03.721556 21366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:03.721724 21366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:03.723767 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "e11bdc5e6257474eaffe225f2d639ea4"
format_stamp: "Formatted at 2026-08-09 04:35:03 on dist-test-slave-rmhg"
server_key: "980eb917380dffd2ed4a1cf43fbfc03e"
server_key_iv: "843899de2b6f90f6fbe8a49587846a8e"
server_key_version: "encryptionkey@0"
I20260809 04:35:03.724327 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "e11bdc5e6257474eaffe225f2d639ea4"
format_stamp: "Formatted at 2026-08-09 04:35:03 on dist-test-slave-rmhg"
server_key: "980eb917380dffd2ed4a1cf43fbfc03e"
server_key_iv: "843899de2b6f90f6fbe8a49587846a8e"
server_key_version: "encryptionkey@0"
I20260809 04:35:03.728278 21366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:35:03.731047 21389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:03.732352 21366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:35:03.732506 21366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "e11bdc5e6257474eaffe225f2d639ea4"
format_stamp: "Formatted at 2026-08-09 04:35:03 on dist-test-slave-rmhg"
server_key: "980eb917380dffd2ed4a1cf43fbfc03e"
server_key_iv: "843899de2b6f90f6fbe8a49587846a8e"
server_key_version: "encryptionkey@0"
I20260809 04:35:03.732613 21366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:03.747118 21366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:03.750028 21366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:03.750221 21366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:03.758466 21366 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:45231
I20260809 04:35:03.758519 21466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:45231 every 8 connection(s)
I20260809 04:35:03.759413 21366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:03.762192 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:03.767927 21467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4: Bootstrap starting.
I20260809 04:35:03.768726 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21366
I20260809 04:35:03.768858 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:03.769142 19824 external_mini_cluster.cc:1428] Setting key b224933d1227d5f8c76036de1595ea14
I20260809 04:35:03.771107 21467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:03.771867 21467 log.cc:824] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:03.774140 21467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4: No bootstrap required, opened a new log
Aug 09 04:35:03 dist-test-slave-rmhg krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:03.778570 21467 raft_consensus.cc:348] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11bdc5e6257474eaffe225f2d639ea4" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45231 } }
I20260809 04:35:03.778865 21467 raft_consensus.cc:374] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:03.778925 21467 raft_consensus.cc:729] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11bdc5e6257474eaffe225f2d639ea4, State: Initialized, Role: FOLLOWER
I20260809 04:35:03.779482 21467 consensus_queue.cc:260] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11bdc5e6257474eaffe225f2d639ea4" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45231 } }
I20260809 04:35:03.779636 21467 raft_consensus.cc:388] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:03.779709 21467 raft_consensus.cc:482] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:03.779820 21467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:03.781024 21467 raft_consensus.cc:504] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11bdc5e6257474eaffe225f2d639ea4" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45231 } }
I20260809 04:35:03.781438 21467 leader_election.cc:302] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e11bdc5e6257474eaffe225f2d639ea4; no voters:
I20260809 04:35:03.781767 21467 leader_election.cc:288] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:03.781852 21475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:03.782069 21475 raft_consensus.cc:686] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [term 1 LEADER]: Becoming Leader. State: Replica: e11bdc5e6257474eaffe225f2d639ea4, State: Running, Role: LEADER
I20260809 04:35:03.782382 21475 consensus_queue.cc:237] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11bdc5e6257474eaffe225f2d639ea4" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45231 } }
I20260809 04:35:03.783171 21467 sys_catalog.cc:564] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:03.784308 21476 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e11bdc5e6257474eaffe225f2d639ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11bdc5e6257474eaffe225f2d639ea4" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45231 } } }
I20260809 04:35:03.784646 21476 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:03.785048 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:03.770553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.19.92.62:45231)
0809 04:35:03.770999 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:03.771015 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:03.771055 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:35:03.771795 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:03.771817 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:03.772206 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:03.772629 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:03.773471 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:03.774531 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:03.775690 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:03.776182 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:03.776302 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:03.779100 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:03.779106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:03.779129 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:03.779165 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:03.781229 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:03.781877 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:03.781884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:03.781891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:03.782062 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:03.782425 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:03.782429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:03.782431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:03.782940 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:03.783140 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:03.783541 (+ 401us) server_negotiation.cc:299] Negotiation successful
0809 04:35:03.783889 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260809 04:35:03.785490 21477 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e11bdc5e6257474eaffe225f2d639ea4. Latest consensus state: current_term: 1 leader_uuid: "e11bdc5e6257474eaffe225f2d639ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11bdc5e6257474eaffe225f2d639ea4" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45231 } } }
I20260809 04:35:03.785588 21477 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:03.786113 21491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:03.788775 21491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:03.794456 21491 catalog_manager.cc:1347] Generated new cluster ID: 21c92b0ee9b24c4fa6893c065c9e96ed
I20260809 04:35:03.794533 21491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:03.803787 21491 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:03.805231 21491 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:03.812785 21491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4: Generated new TSK 0
I20260809 04:35:03.813601 21491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:03.852145 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:45231
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:03.968765 21505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:03.969079 21505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:03.969125 21505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:03.973208 21505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:03.973281 21505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:03.973320 21505 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:03.973353 21505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:03.978718 21505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:03.979837 21505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:03.980856 21505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:03.990429 21515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:03.990440 21513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:03.990440 21512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:03.990501 21505 server_base.cc:1034] running on GCE node
I20260809 04:35:03.991024 21505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:03.991636 21505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:03.992798 21505 hybrid_clock.cc:648] HybridClock initialized: now 1786250103992773 us; error 36 us; skew 500 ppm
Aug 09 04:35:03 dist-test-slave-rmhg krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250103, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:03.995544 21505 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:03.996419 21505 webserver.cc:460] Webserver started at http://127.19.92.1:40523/ using document root <none> and password file <none>
I20260809 04:35:03.996969 21505 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:03.997031 21505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:03.997278 21505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:03.999333 21505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "8f6542b7192446f798840c60e86db31d"
format_stamp: "Formatted at 2026-08-09 04:35:03 on dist-test-slave-rmhg"
server_key: "31e347188b04a543c34de6386528cac3"
server_key_iv: "89d236076d7c789612477ec6b6854115"
server_key_version: "encryptionkey@0"
I20260809 04:35:03.999897 21505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "8f6542b7192446f798840c60e86db31d"
format_stamp: "Formatted at 2026-08-09 04:35:03 on dist-test-slave-rmhg"
server_key: "31e347188b04a543c34de6386528cac3"
server_key_iv: "89d236076d7c789612477ec6b6854115"
server_key_version: "encryptionkey@0"
I20260809 04:35:04.003777 21505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:35:04.006217 21524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:04.007794 21505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:04.007906 21505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "8f6542b7192446f798840c60e86db31d"
format_stamp: "Formatted at 2026-08-09 04:35:03 on dist-test-slave-rmhg"
server_key: "31e347188b04a543c34de6386528cac3"
server_key_iv: "89d236076d7c789612477ec6b6854115"
server_key_version: "encryptionkey@0"
I20260809 04:35:04.008023 21505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:04.035593 21505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:04.038585 21505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:04.038762 21505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:04.040271 21505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:04.040323 21505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:04.040364 21505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:04.040392 21505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:35:04 dist-test-slave-rmhg krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250103, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:04.050282 21505 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:45137
I20260809 04:35:04.050477 21688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:45137 every 8 connection(s)
I20260809 04:35:04.051419 21505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:04.053812 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.041030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:42397 (local address 127.19.92.62:45231)
0809 04:35:04.041194 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.041198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.042085 (+ 887us) server_negotiation.cc:407] Connection header received
0809 04:35:04.043195 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.043199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.043260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.043344 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:04.044815 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.045684 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.047003 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.047253 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.047349 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.050037 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:04.050041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:04.050044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:04.050076 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:04.051756 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.052345 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.052348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.052350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.052449 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.052759 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.052762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.052763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.052946 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:04.053019 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.053553 (+ 534us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.053683 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:35:04.054569 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.041462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45231 (local address 127.19.92.1:42397)
0809 04:35:04.041914 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.041964 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.042905 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.043555 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.043565 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.043975 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:04.044657 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.044679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.045824 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.045828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.046882 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.046892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.047407 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.047411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.047590 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.048143 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:04.048164 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:04.049801 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:04.051881 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.051887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.051902 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.052237 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.052548 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.052551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.052553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.052685 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.053029 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:04.053034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:04.053299 (+ 265us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.053456 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.053717 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260809 04:35:04.055869 21689 heartbeater.cc:347] Connected to a master server at 127.19.92.62:45231
I20260809 04:35:04.056022 21689 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:04.056521 21689 heartbeater.cc:511] Master 127.19.92.62:45231 requested a full tablet report, sending...
I20260809 04:35:04.058008 21408 ts_manager.cc:194] Registered new tserver with Master: 8f6542b7192446f798840c60e86db31d (127.19.92.1:45137)
I20260809 04:35:04.059540 21408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:42397
I20260809 04:35:04.060722 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21505
I20260809 04:35:04.060844 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:04.061192 19824 external_mini_cluster.cc:1428] Setting key 1bc96d32a12e8f69e967cc124f02e0e9
I20260809 04:35:04.065115 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.058044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:50663 (local address 127.19.92.62:45231)
0809 04:35:04.058169 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.058173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.058269 (+ 96us) server_negotiation.cc:407] Connection header received
0809 04:35:04.058364 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.058367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.058408 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.058480 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:04.058873 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.059634 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.060433 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.060738 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.060840 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.062171 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:04.062175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:04.062178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:04.062205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:04.063523 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.063964 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.063967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.063968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.064049 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.064291 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.064292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.064294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.064512 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:04.064586 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.064826 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.064952 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:35:04.065135 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.057948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45231 (local address 127.19.92.1:50663)
0809 04:35:04.058197 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.058209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.058288 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.058507 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.058509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.058669 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:04.058768 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.058773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.059751 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.059754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.060323 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.060331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.060967 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.060971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.061085 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.061625 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:04.061639 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:04.062035 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:04.063627 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.063631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.063633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.063866 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.064123 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.064126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.064128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.064217 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.064591 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:04.064594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:04.064689 (+ 95us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.064792 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.064933 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:04.089908 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:45231
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:04.202865 21703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:04.203120 21703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:04.203163 21703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:04.207096 21703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:04.207152 21703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:04.207190 21703 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:04.207219 21703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:04.212940 21703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:04.214057 21703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:04.215059 21703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:04.224741 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:04.224749 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:04.224749 21720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:04.224786 21703 server_base.cc:1034] running on GCE node
I20260809 04:35:04.225394 21703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:04.226086 21703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:04.227259 21703 hybrid_clock.cc:648] HybridClock initialized: now 1786250104227237 us; error 35 us; skew 500 ppm
Aug 09 04:35:04 dist-test-slave-rmhg krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250104, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:04.229979 21703 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:04.230855 21703 webserver.cc:460] Webserver started at http://127.19.92.2:35771/ using document root <none> and password file <none>
I20260809 04:35:04.231379 21703 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:04.231456 21703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:04.231815 21703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:04.233973 21703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "38a4d63d7d99429f994f1e9d0a95f639"
format_stamp: "Formatted at 2026-08-09 04:35:04 on dist-test-slave-rmhg"
server_key: "f5a6b348556a15cf60ee249195187e9f"
server_key_iv: "4360fbbd4ac23daaef31c78ace232f37"
server_key_version: "encryptionkey@0"
I20260809 04:35:04.234513 21703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "38a4d63d7d99429f994f1e9d0a95f639"
format_stamp: "Formatted at 2026-08-09 04:35:04 on dist-test-slave-rmhg"
server_key: "f5a6b348556a15cf60ee249195187e9f"
server_key_iv: "4360fbbd4ac23daaef31c78ace232f37"
server_key_version: "encryptionkey@0"
I20260809 04:35:04.238243 21703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:35:04.240617 21729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:04.241858 21703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:04.241967 21703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "38a4d63d7d99429f994f1e9d0a95f639"
format_stamp: "Formatted at 2026-08-09 04:35:04 on dist-test-slave-rmhg"
server_key: "f5a6b348556a15cf60ee249195187e9f"
server_key_iv: "4360fbbd4ac23daaef31c78ace232f37"
server_key_version: "encryptionkey@0"
I20260809 04:35:04.242063 21703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:04.264202 21703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:04.267561 21703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:04.267768 21703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:04.269645 21703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:04.269699 21703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:04.269752 21703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:04.269781 21703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:35:04 dist-test-slave-rmhg krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250104, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:04.280779 21703 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:44433
I20260809 04:35:04.280851 21890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:44433 every 8 connection(s)
I20260809 04:35:04.281742 21703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:04.282528 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.270370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:51961 (local address 127.19.92.62:45231)
0809 04:35:04.270518 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.270523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.271344 (+ 821us) server_negotiation.cc:407] Connection header received
0809 04:35:04.272299 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.272303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.272360 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.272416 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:04.274051 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.274631 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.275760 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.276094 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.276187 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.278934 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:04.278938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:04.278940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:04.278971 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:04.280460 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.281016 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.281019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.281020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.281098 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.281437 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.281441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.281443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.281629 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:04.281718 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.282240 (+ 522us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.282362 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:35:04.283272 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.270734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45231 (local address 127.19.92.2:51961)
0809 04:35:04.271172 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.271223 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.272069 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.272857 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.272866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.273343 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:04.273910 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.273931 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.274754 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.274758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.275641 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.275650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.276260 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.276265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.276442 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.276997 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:04.277020 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:04.278771 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:04.280562 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.280568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.280580 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.280866 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.281209 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.281212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.281214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.281348 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.281729 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:04.281735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:04.282003 (+ 268us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.282163 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.282489 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260809 04:35:04.284493 21892 heartbeater.cc:347] Connected to a master server at 127.19.92.62:45231
I20260809 04:35:04.284629 21892 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:04.285094 21892 heartbeater.cc:511] Master 127.19.92.62:45231 requested a full tablet report, sending...
I20260809 04:35:04.286185 21410 ts_manager.cc:194] Registered new tserver with Master: 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433)
I20260809 04:35:04.286873 21410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:51961
I20260809 04:35:04.287612 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21703
I20260809 04:35:04.287711 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:04.287925 19824 external_mini_cluster.cc:1428] Setting key df8c99627f403fe54ac40ebbbf3254b5
I20260809 04:35:04.294196 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.286778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45231 (local address 127.19.92.2:33079)
0809 04:35:04.287032 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.287043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.287246 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.287507 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.287510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.287732 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:04.287859 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.287867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.288765 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.288768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.289677 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.289690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.290135 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.290139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.290231 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.290714 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:04.290728 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:04.291144 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:04.292665 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.292669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.292671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.292916 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.293212 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.293215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.293217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.293304 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.293762 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:04.293765 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:04.293849 (+ 84us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.293933 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.294053 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260809 04:35:04.294269 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.286837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:33079 (local address 127.19.92.62:45231)
0809 04:35:04.286946 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.286950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.287109 (+ 159us) server_negotiation.cc:407] Connection header received
0809 04:35:04.287349 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.287352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.287399 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.287509 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:04.287985 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.288639 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.289797 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.290027 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.290109 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.291282 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:04.291285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:04.291287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:04.291309 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:04.292557 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.293013 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.293016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.293018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.293104 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.293380 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.293383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.293385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.293659 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:04.293752 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.294007 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.294119 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:04.316509 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:45231
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:04.434541 21903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:04.434820 21903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:04.434886 21903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:04.439041 21903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:04.439111 21903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:04.439168 21903 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:04.439224 21903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:04.444746 21903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:04.445996 21903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:04.447067 21903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:04.457617 21916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:04.457617 21908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:04.457631 21909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:04.457971 21903 server_base.cc:1034] running on GCE node
I20260809 04:35:04.458451 21903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:04.459161 21903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:04.460343 21903 hybrid_clock.cc:648] HybridClock initialized: now 1786250104460324 us; error 32 us; skew 500 ppm
Aug 09 04:35:04 dist-test-slave-rmhg krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250104, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:04.463384 21903 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:04.464370 21903 webserver.cc:460] Webserver started at http://127.19.92.3:42325/ using document root <none> and password file <none>
I20260809 04:35:04.464972 21903 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:04.465044 21903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:04.465307 21903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:04.467530 21903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "e151f99fd6304a89aa592e844dfba9e2"
format_stamp: "Formatted at 2026-08-09 04:35:04 on dist-test-slave-rmhg"
server_key: "e59977acd4434c43d477025398e1a11b"
server_key_iv: "a5742197d0475cb23d3db47078e54b3c"
server_key_version: "encryptionkey@0"
I20260809 04:35:04.468127 21903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "e151f99fd6304a89aa592e844dfba9e2"
format_stamp: "Formatted at 2026-08-09 04:35:04 on dist-test-slave-rmhg"
server_key: "e59977acd4434c43d477025398e1a11b"
server_key_iv: "a5742197d0475cb23d3db47078e54b3c"
server_key_version: "encryptionkey@0"
I20260809 04:35:04.472157 21903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:35:04.474823 21924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:04.476226 21903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:35:04.476410 21903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "e151f99fd6304a89aa592e844dfba9e2"
format_stamp: "Formatted at 2026-08-09 04:35:04 on dist-test-slave-rmhg"
server_key: "e59977acd4434c43d477025398e1a11b"
server_key_iv: "a5742197d0475cb23d3db47078e54b3c"
server_key_version: "encryptionkey@0"
I20260809 04:35:04.476537 21903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:04.515429 21903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:04.518610 21903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:04.518798 21903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:04.520326 21903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:04.520392 21903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:04.520450 21903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:04.520475 21903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:35:04 dist-test-slave-rmhg krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250104, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:04.531553 21903 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:40601
I20260809 04:35:04.531891 22094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:40601 every 8 connection(s)
I20260809 04:35:04.532591 21903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:04.534132 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21903
I20260809 04:35:04.534252 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:04.534535 19824 external_mini_cluster.cc:1428] Setting key cfb35d86fe696669fe5d2879b2cb8b31
I20260809 04:35:04.534891 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.521308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:39309 (local address 127.19.92.62:45231)
0809 04:35:04.521425 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.521429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.522521 (+ 1092us) server_negotiation.cc:407] Connection header received
0809 04:35:04.523645 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.523649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.523715 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.523776 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:04.525598 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.526226 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.527551 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.527858 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.527962 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.530874 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:04.530877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:04.530879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:04.530911 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:04.532349 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.532937 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.532940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.532941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.533047 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.533644 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.533647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.533648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.533902 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:04.534012 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.534589 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.534732 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:35:04.535810 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.521846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45231 (local address 127.19.92.3:39309)
0809 04:35:04.522361 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.522407 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.523380 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.524224 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.524234 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.524680 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:04.525404 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.525422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.526361 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.526364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.527420 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.527431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.528030 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.528035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.528228 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.528863 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:04.528881 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:04.530659 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:04.532481 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.532488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.532504 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.532825 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.533313 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.533317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.533320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.533518 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.534019 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:04.534024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:04.534297 (+ 273us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.534476 (+ 179us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.534791 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260809 04:35:04.537717 22095 heartbeater.cc:347] Connected to a master server at 127.19.92.62:45231
I20260809 04:35:04.537874 22095 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:04.538398 22095 heartbeater.cc:511] Master 127.19.92.62:45231 requested a full tablet report, sending...
I20260809 04:35:04.539479 21408 ts_manager.cc:194] Registered new tserver with Master: e151f99fd6304a89aa592e844dfba9e2 (127.19.92.3:40601)
I20260809 04:35:04.540205 21408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:39309
I20260809 04:35:04.546557 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.539867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:50023 (local address 127.19.92.62:45231)
0809 04:35:04.540002 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.540005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.540235 (+ 230us) server_negotiation.cc:407] Connection header received
0809 04:35:04.540324 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.540327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.540369 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.540459 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:04.540942 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.541715 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.542571 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.542783 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.542861 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.543836 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:04.543839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:04.543841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:04.543861 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:04.545245 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.545604 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.545606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.545607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.545674 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.545868 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.545870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.545871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.546040 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:04.546125 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.546358 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.546448 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:35:04.546568 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.539775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45231 (local address 127.19.92.3:50023)
0809 04:35:04.540140 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.540151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.540247 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.540480 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.540512 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.540714 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:04.540822 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.540827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.541848 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.541852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.542429 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.542437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.542880 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.542883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.542968 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.543380 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:04.543390 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:04.543718 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:04.545337 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.545340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.545341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.545532 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.545741 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:04.545743 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:04.545744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:04.545811 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:04.546149 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:04.546152 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:04.546231 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.546314 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.546425 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260809 04:35:04.548645 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:35:04 dist-test-slave-rmhg krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:35:04 dist-test-slave-rmhg krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:04.575366 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.566886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.19.92.62:45231)
0809 04:35:04.567022 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.567026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.567039 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:04.567082 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.567085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.567127 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.567199 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:04.567906 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.568477 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.569435 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.569676 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.569777 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.572352 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:04.572356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:04.572358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:04.572393 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:04.573812 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.574251 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.574254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.574255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.574333 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:04.574545 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:04.574547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:04.574548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:04.574745 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:04.574849 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.575107 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.575227 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:35:04.578755 21408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:04.582034 21408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:04.597434 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.591842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.19.92.1:45137)
0809 04:35:04.592672 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.592678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.592698 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:35:04.592761 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.592764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.592919 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.593215 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:04.593632 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.594763 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.595960 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.596690 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.596796 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.596965 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.597116 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.597276 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":80,"threads_started":1}
I20260809 04:35:04.597434 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.591711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45137 (local address 127.0.0.1:35406)
0809 04:35:04.592288 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.592302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.592404 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.593360 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.593363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.593379 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:04.593482 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.593489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.594879 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.594883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.595832 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.595842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.596808 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.596812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.596926 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.596965 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.597075 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.597286 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":28,"spinlock_wait_cycles":7168,"thread_start_us":109,"threads_started":1}
I20260809 04:35:04.598073 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.592064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:44433 (local address 127.0.0.1:54148)
0809 04:35:04.592317 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.592330 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.592404 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.593447 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.593449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.593463 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:04.593529 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.593535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.594631 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.594634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.596133 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.596144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.597472 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.597475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.597565 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.597788 (+ 223us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.597864 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.597944 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":63,"spinlock_wait_cycles":3712,"thread_start_us":81,"threads_started":1}
I20260809 04:35:04.598138 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.592319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.19.92.2:44433)
0809 04:35:04.592672 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.592684 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.592702 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:35:04.592761 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.592766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.592940 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.593101 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:04.593628 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.594510 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.596275 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.597112 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.597237 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.597402 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.597899 (+ 497us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.597999 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260809 04:35:04.598645 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.590944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:40601 (local address 127.0.0.1:45080)
0809 04:35:04.592183 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.592241 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.592382 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.593115 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.593122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.593175 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:04.593324 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.593332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.594808 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.594812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.596152 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.596159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.597273 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.597276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.597368 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.598427 (+ 1059us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.598489 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.598550 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":81,"thread_start_us":78,"threads_started":1}
I20260809 04:35:04.599332 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.591146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.19.92.3:40601)
0809 04:35:04.591543 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.591558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.592624 (+ 1066us) server_negotiation.cc:407] Connection header received
0809 04:35:04.592721 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.592727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.592940 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.593101 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:04.593435 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.594669 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.596275 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.597113 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.598896 (+ 1783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.599036 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.599093 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.599191 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":144,"threads_started":1}
I20260809 04:35:04.600672 21593 tablet_service.cc:1467] Processing CreateTablet for tablet bcc1005dcd644aa3a84ad671b33aecb3 (DEFAULT_TABLE table=test-table [id=8b0b210aa0f54593a2d3cb444dff46d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:04.600999 21796 tablet_service.cc:1467] Processing CreateTablet for tablet bcc1005dcd644aa3a84ad671b33aecb3 (DEFAULT_TABLE table=test-table [id=8b0b210aa0f54593a2d3cb444dff46d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:04.601961 21796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc1005dcd644aa3a84ad671b33aecb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:04.601963 21593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc1005dcd644aa3a84ad671b33aecb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:04.602144 21993 tablet_service.cc:1467] Processing CreateTablet for tablet bcc1005dcd644aa3a84ad671b33aecb3 (DEFAULT_TABLE table=test-table [id=8b0b210aa0f54593a2d3cb444dff46d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:04.603171 21993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc1005dcd644aa3a84ad671b33aecb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:04.609633 22130 tablet_bootstrap.cc:492] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Bootstrap starting.
I20260809 04:35:04.610302 22131 tablet_bootstrap.cc:492] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Bootstrap starting.
I20260809 04:35:04.611263 22132 tablet_bootstrap.cc:492] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Bootstrap starting.
I20260809 04:35:04.612227 22130 tablet_bootstrap.cc:654] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:04.612641 22131 tablet_bootstrap.cc:654] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:04.613217 22130 log.cc:824] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:04.613415 22131 log.cc:824] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:04.613631 22132 tablet_bootstrap.cc:654] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:04.614461 22132 log.cc:824] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:04.615393 22130 tablet_bootstrap.cc:492] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: No bootstrap required, opened a new log
I20260809 04:35:04.615597 22130 ts_tablet_manager.cc:1397] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:35:04.616014 22131 tablet_bootstrap.cc:492] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: No bootstrap required, opened a new log
I20260809 04:35:04.616209 22131 ts_tablet_manager.cc:1397] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:35:04.616745 22132 tablet_bootstrap.cc:492] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: No bootstrap required, opened a new log
I20260809 04:35:04.616923 22132 ts_tablet_manager.cc:1397] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:35:04.619261 22130 raft_consensus.cc:348] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.619503 22130 raft_consensus.cc:374] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:04.619555 22130 raft_consensus.cc:729] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6542b7192446f798840c60e86db31d, State: Initialized, Role: FOLLOWER
I20260809 04:35:04.620018 22130 consensus_queue.cc:260] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.620174 22131 raft_consensus.cc:348] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.620455 22131 raft_consensus.cc:374] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:04.620517 22131 raft_consensus.cc:729] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a4d63d7d99429f994f1e9d0a95f639, State: Initialized, Role: FOLLOWER
I20260809 04:35:04.620824 21689 heartbeater.cc:503] Master 127.19.92.62:45231 was elected leader, sending a full tablet report...
I20260809 04:35:04.620785 22132 raft_consensus.cc:348] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.621024 22130 ts_tablet_manager.cc:1428] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 04:35:04.621066 22132 raft_consensus.cc:374] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:04.621125 22132 raft_consensus.cc:729] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e151f99fd6304a89aa592e844dfba9e2, State: Initialized, Role: FOLLOWER
I20260809 04:35:04.621580 22132 consensus_queue.cc:260] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.622241 22095 heartbeater.cc:503] Master 127.19.92.62:45231 was elected leader, sending a full tablet report...
I20260809 04:35:04.622181 22131 consensus_queue.cc:260] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.623217 22132 ts_tablet_manager.cc:1428] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260809 04:35:04.623263 22131 ts_tablet_manager.cc:1428] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 04:35:04.623718 21892 heartbeater.cc:503] Master 127.19.92.62:45231 was elected leader, sending a full tablet report...
W20260809 04:35:04.783655 21894 tablet.cc:2369] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:35:04.783756 22096 tablet.cc:2369] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:35:04.803277 21691 tablet.cc:2369] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:35:04.915380 22140 raft_consensus.cc:482] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:04.915661 22140 raft_consensus.cc:504] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.916846 22140 leader_election.cc:288] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f6542b7192446f798840c60e86db31d (127.19.92.1:45137), 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433)
I20260809 04:35:04.920804 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.917241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45137 (local address 127.19.92.3:47463)
0809 04:35:04.917374 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.917390 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.917505 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.917856 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.917859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.917883 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:04.918044 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.918051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.918814 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.918816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.919773 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.919784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.920467 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.920470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.920556 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.920578 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.920635 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.920693 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 04:35:04.920863 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.917332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:47463 (local address 127.19.92.1:45137)
0809 04:35:04.917461 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.917465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.917485 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:35:04.917630 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.917634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.917688 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.917781 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:04.918152 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.918694 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.919900 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.920350 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.920456 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.920524 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.920666 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.920738 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:35:04.921547 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.917693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:39889 (local address 127.19.92.2:44433)
0809 04:35:04.917818 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.917822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.917940 (+ 118us) server_negotiation.cc:407] Connection header received
0809 04:35:04.918016 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.918018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.918060 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.918141 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:04.918787 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.919297 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.920465 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.921021 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.921128 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.921212 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.921393 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.921455 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:35:04.921556 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.917528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:44433 (local address 127.19.92.3:39889)
0809 04:35:04.917838 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:04.917851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:04.917938 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:04.918157 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:04.918159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:04.918174 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:04.918658 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.918666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.919405 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.919409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.920298 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:04.920307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.921140 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.921144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:04.921266 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.921292 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 04:35:04.921366 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:35:04.921430 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
I20260809 04:35:04.921555 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc1005dcd644aa3a84ad671b33aecb3" candidate_uuid: "e151f99fd6304a89aa592e844dfba9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6542b7192446f798840c60e86db31d" is_pre_election: true
I20260809 04:35:04.921907 21619 raft_consensus.cc:2454] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e151f99fd6304a89aa592e844dfba9e2 in term 0.
I20260809 04:35:04.922116 21819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc1005dcd644aa3a84ad671b33aecb3" candidate_uuid: "e151f99fd6304a89aa592e844dfba9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a4d63d7d99429f994f1e9d0a95f639" is_pre_election: true
I20260809 04:35:04.922415 21819 raft_consensus.cc:2454] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e151f99fd6304a89aa592e844dfba9e2 in term 0.
I20260809 04:35:04.922431 21927 leader_election.cc:302] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f6542b7192446f798840c60e86db31d, e151f99fd6304a89aa592e844dfba9e2; no voters:
I20260809 04:35:04.922748 22140 raft_consensus.cc:2781] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:04.922853 22140 raft_consensus.cc:482] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:04.922909 22140 raft_consensus.cc:3037] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:04.924021 22140 raft_consensus.cc:504] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.924351 22140 leader_election.cc:288] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [CANDIDATE]: Term 1 election: Requested vote from peers 8f6542b7192446f798840c60e86db31d (127.19.92.1:45137), 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433)
I20260809 04:35:04.924800 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc1005dcd644aa3a84ad671b33aecb3" candidate_uuid: "e151f99fd6304a89aa592e844dfba9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6542b7192446f798840c60e86db31d"
I20260809 04:35:04.924849 21819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc1005dcd644aa3a84ad671b33aecb3" candidate_uuid: "e151f99fd6304a89aa592e844dfba9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a4d63d7d99429f994f1e9d0a95f639"
I20260809 04:35:04.924944 21619 raft_consensus.cc:3037] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:04.924978 21819 raft_consensus.cc:3037] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:04.926316 21819 raft_consensus.cc:2454] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e151f99fd6304a89aa592e844dfba9e2 in term 1.
I20260809 04:35:04.926316 21619 raft_consensus.cc:2454] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e151f99fd6304a89aa592e844dfba9e2 in term 1.
I20260809 04:35:04.926681 21927 leader_election.cc:302] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a4d63d7d99429f994f1e9d0a95f639, e151f99fd6304a89aa592e844dfba9e2; no voters:
I20260809 04:35:04.926862 22140 raft_consensus.cc:2781] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:04.927171 22140 raft_consensus.cc:686] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 1 LEADER]: Becoming Leader. State: Replica: e151f99fd6304a89aa592e844dfba9e2, State: Running, Role: LEADER
I20260809 04:35:04.927619 22140 consensus_queue.cc:237] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } }
I20260809 04:35:04.931053 21407 catalog_manager.cc:5495] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e151f99fd6304a89aa592e844dfba9e2 (127.19.92.3). New cstate: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } health_report { overall_health: UNKNOWN } } }
I20260809 04:35:04.968194 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:04.964365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.19.92.3:40601)
0809 04:35:04.964568 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:04.964573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:04.964593 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:35:04.964698 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:04.964701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:04.964760 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:04.964856 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:04.965210 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.965848 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.966784 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:04.966992 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:04.967065 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:04.967371 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:04.967807 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:04.967891 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:04.968018 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 04:35:04.968076 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260809 04:35:04.974550 21819 raft_consensus.cc:1261] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Refusing update from remote peer e151f99fd6304a89aa592e844dfba9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:04.974550 21619 raft_consensus.cc:1261] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Refusing update from remote peer e151f99fd6304a89aa592e844dfba9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:04.975327 22143 consensus_queue.cc:1035] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:04.975533 22140 consensus_queue.cc:1035] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:04.983472 22147 mvcc.cc:205] Tried to move back new op lower bound from 7316480429967405056 to 7316480429786423296. Current Snapshot: MvccSnapshot[applied={T|T < 7316480429967405056}]
I20260809 04:35:04.983956 22152 mvcc.cc:205] Tried to move back new op lower bound from 7316480429967405056 to 7316480429786423296. Current Snapshot: MvccSnapshot[applied={T|T < 7316480429967405056}]
I20260809 04:35:04.987563 22151 mvcc.cc:205] Tried to move back new op lower bound from 7316480429967405056 to 7316480429786423296. Current Snapshot: MvccSnapshot[applied={T|T < 7316480429967405056}]
I20260809 04:35:04.996318 21408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:04.996599 21408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:04.999552 21408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4: Sending DeleteTablet for 3 replicas of tablet bcc1005dcd644aa3a84ad671b33aecb3
I20260809 04:35:05.000388 21993 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc1005dcd644aa3a84ad671b33aecb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:04 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:45080
I20260809 04:35:05.000552 21593 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc1005dcd644aa3a84ad671b33aecb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:04 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:35406
I20260809 04:35:05.000655 21796 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc1005dcd644aa3a84ad671b33aecb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:04 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:54148
I20260809 04:35:05.000977 22162 tablet_replica.cc:330] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: stopping tablet replica
I20260809 04:35:05.001242 21408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 04:35:05.001343 22164 tablet_replica.cc:330] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: stopping tablet replica
I20260809 04:35:05.001417 22163 tablet_replica.cc:330] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: stopping tablet replica
W20260809 04:35:05.001693 21408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:05.001816 22163 raft_consensus.cc:2229] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:05.002038 22164 raft_consensus.cc:2229] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:05.002192 22163 raft_consensus.cc:2258] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:05.002480 22164 raft_consensus.cc:2258] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:05.002684 22162 raft_consensus.cc:2229] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:05.003130 22162 raft_consensus.cc:2258] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:05.004679 22163 ts_tablet_manager.cc:1905] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:05.004709 22162 ts_tablet_manager.cc:1905] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:05.007795 22162 ts_tablet_manager.cc:1918] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:05.008013 22162 log.cc:1196] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/bcc1005dcd644aa3a84ad671b33aecb3
I20260809 04:35:05.008013 21593 tablet_service.cc:1467] Processing CreateTablet for tablet ff0c7b1640484c22a89b3faf67dd4f63 (DEFAULT_TABLE table=test-table [id=8456c8b6959b4c148f45fd9d72eacb61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:05.008267 21993 tablet_service.cc:1467] Processing CreateTablet for tablet ff0c7b1640484c22a89b3faf67dd4f63 (DEFAULT_TABLE table=test-table [id=8456c8b6959b4c148f45fd9d72eacb61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:05.008319 21593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff0c7b1640484c22a89b3faf67dd4f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:05.008524 21993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff0c7b1640484c22a89b3faf67dd4f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:05.008671 22162 ts_tablet_manager.cc:1939] T bcc1005dcd644aa3a84ad671b33aecb3 P e151f99fd6304a89aa592e844dfba9e2: Deleting consensus metadata
I20260809 04:35:05.011044 22130 tablet_bootstrap.cc:492] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: Bootstrap starting.
I20260809 04:35:05.011564 21796 tablet_service.cc:1467] Processing CreateTablet for tablet ff0c7b1640484c22a89b3faf67dd4f63 (DEFAULT_TABLE table=test-table [id=8456c8b6959b4c148f45fd9d72eacb61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:05.012497 21796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff0c7b1640484c22a89b3faf67dd4f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:05.011351 21392 catalog_manager.cc:4841] TS e151f99fd6304a89aa592e844dfba9e2 (127.19.92.3:40601): tablet bcc1005dcd644aa3a84ad671b33aecb3 (table test-table [id=8b0b210aa0f54593a2d3cb444dff46d8]) successfully deleted
I20260809 04:35:05.012724 22130 tablet_bootstrap.cc:654] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:05.008023 22163 ts_tablet_manager.cc:1918] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:05.012980 22163 log.cc:1196] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/bcc1005dcd644aa3a84ad671b33aecb3
I20260809 04:35:05.013360 22163 ts_tablet_manager.cc:1939] T bcc1005dcd644aa3a84ad671b33aecb3 P 8f6542b7192446f798840c60e86db31d: Deleting consensus metadata
I20260809 04:35:05.014748 21392 catalog_manager.cc:4841] TS 8f6542b7192446f798840c60e86db31d (127.19.92.1:45137): tablet bcc1005dcd644aa3a84ad671b33aecb3 (table test-table [id=8b0b210aa0f54593a2d3cb444dff46d8]) successfully deleted
I20260809 04:35:05.015654 22131 tablet_bootstrap.cc:492] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: Bootstrap starting.
I20260809 04:35:05.015872 22130 tablet_bootstrap.cc:492] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: No bootstrap required, opened a new log
I20260809 04:35:05.016001 22130 ts_tablet_manager.cc:1397] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 04:35:05.016738 22130 raft_consensus.cc:348] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.016930 22164 ts_tablet_manager.cc:1905] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:05.016960 22130 raft_consensus.cc:374] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:05.017071 22130 raft_consensus.cc:729] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6542b7192446f798840c60e86db31d, State: Initialized, Role: FOLLOWER
I20260809 04:35:05.017374 22131 tablet_bootstrap.cc:654] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:05.017324 22130 consensus_queue.cc:260] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.017808 22130 ts_tablet_manager.cc:1428] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:05.019225 22132 tablet_bootstrap.cc:492] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: Bootstrap starting.
I20260809 04:35:05.020522 22164 ts_tablet_manager.cc:1918] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:05.020632 22164 log.cc:1196] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/bcc1005dcd644aa3a84ad671b33aecb3
I20260809 04:35:05.020955 22164 ts_tablet_manager.cc:1939] T bcc1005dcd644aa3a84ad671b33aecb3 P 38a4d63d7d99429f994f1e9d0a95f639: Deleting consensus metadata
I20260809 04:35:05.021070 22132 tablet_bootstrap.cc:654] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:05.022380 21392 catalog_manager.cc:4841] TS 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433): tablet bcc1005dcd644aa3a84ad671b33aecb3 (table test-table [id=8b0b210aa0f54593a2d3cb444dff46d8]) successfully deleted
I20260809 04:35:05.023236 22131 tablet_bootstrap.cc:492] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: No bootstrap required, opened a new log
I20260809 04:35:05.023327 22131 ts_tablet_manager.cc:1397] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260809 04:35:05.023499 22132 tablet_bootstrap.cc:492] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: No bootstrap required, opened a new log
I20260809 04:35:05.023599 22132 ts_tablet_manager.cc:1397] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260809 04:35:05.023973 22131 raft_consensus.cc:348] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.024072 22131 raft_consensus.cc:374] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:05.024101 22131 raft_consensus.cc:729] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a4d63d7d99429f994f1e9d0a95f639, State: Initialized, Role: FOLLOWER
I20260809 04:35:05.024272 22131 consensus_queue.cc:260] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.024401 22132 raft_consensus.cc:348] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.024555 22132 raft_consensus.cc:374] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:05.024592 22132 raft_consensus.cc:729] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e151f99fd6304a89aa592e844dfba9e2, State: Initialized, Role: FOLLOWER
I20260809 04:35:05.024752 22132 consensus_queue.cc:260] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.024993 22132 ts_tablet_manager.cc:1428] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:05.025336 22131 ts_tablet_manager.cc:1428] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 04:35:05.102618 22138 raft_consensus.cc:482] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:05.102833 22138 raft_consensus.cc:504] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.104012 22138 leader_election.cc:288] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433), e151f99fd6304a89aa592e844dfba9e2 (127.19.92.3:40601)
I20260809 04:35:05.110090 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.104321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:49785 (local address 127.19.92.2:44433)
0809 04:35:05.104509 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.104514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.105250 (+ 736us) server_negotiation.cc:407] Connection header received
0809 04:35:05.105394 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.105398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.105457 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.105555 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:05.105850 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.106656 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.108825 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.109518 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.109634 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.109699 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.109892 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.109982 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:35:05.110198 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.104769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:52597 (local address 127.19.92.3:40601)
0809 04:35:05.104906 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.104911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.105284 (+ 373us) server_negotiation.cc:407] Connection header received
0809 04:35:05.105462 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.105465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.105518 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.105586 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:05.106015 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.106728 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.108786 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.109294 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.109365 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.109433 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.109979 (+ 546us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.110071 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:05.110451 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.104246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:44433 (local address 127.19.92.1:49785)
0809 04:35:05.105097 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:05.105113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:05.105280 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:05.105584 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:05.105587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:05.105605 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:05.105711 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.105718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.106810 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.106813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.108638 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.108648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.109643 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.109646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.109751 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.109778 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 04:35:05.109855 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:35:05.109937 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":5,"thread_start_us":100,"threads_started":1}
I20260809 04:35:05.110659 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.104638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:40601 (local address 127.19.92.1:52597)
0809 04:35:05.105214 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:05.105226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:05.105340 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:05.105692 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:05.105695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:05.105708 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:05.105885 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.105894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.106889 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.106893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.108478 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.108607 (+ 129us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.109522 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.109524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.109714 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.109827 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 04:35:05.110404 (+ 577us) client_negotiation.cc:240] Negotiation successful
0809 04:35:05.110490 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":61,"thread_start_us":190,"threads_started":1}
I20260809 04:35:05.110960 21819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0c7b1640484c22a89b3faf67dd4f63" candidate_uuid: "8f6542b7192446f798840c60e86db31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a4d63d7d99429f994f1e9d0a95f639" is_pre_election: true
I20260809 04:35:05.111127 21819 raft_consensus.cc:2454] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6542b7192446f798840c60e86db31d in term 0.
I20260809 04:35:05.111234 22023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0c7b1640484c22a89b3faf67dd4f63" candidate_uuid: "8f6542b7192446f798840c60e86db31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e151f99fd6304a89aa592e844dfba9e2" is_pre_election: true
I20260809 04:35:05.111485 22023 raft_consensus.cc:2454] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6542b7192446f798840c60e86db31d in term 0.
I20260809 04:35:05.111855 21529 leader_election.cc:302] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a4d63d7d99429f994f1e9d0a95f639, 8f6542b7192446f798840c60e86db31d; no voters:
I20260809 04:35:05.112145 22138 raft_consensus.cc:2781] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:05.112213 22138 raft_consensus.cc:482] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:05.112257 22138 raft_consensus.cc:3037] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:05.113914 22138 raft_consensus.cc:504] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.114297 22138 leader_election.cc:288] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [CANDIDATE]: Term 1 election: Requested vote from peers 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433), e151f99fd6304a89aa592e844dfba9e2 (127.19.92.3:40601)
I20260809 04:35:05.114758 21819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0c7b1640484c22a89b3faf67dd4f63" candidate_uuid: "8f6542b7192446f798840c60e86db31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a4d63d7d99429f994f1e9d0a95f639"
I20260809 04:35:05.114785 22023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff0c7b1640484c22a89b3faf67dd4f63" candidate_uuid: "8f6542b7192446f798840c60e86db31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e151f99fd6304a89aa592e844dfba9e2"
I20260809 04:35:05.114879 21819 raft_consensus.cc:3037] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:05.114928 22023 raft_consensus.cc:3037] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:05.115985 21819 raft_consensus.cc:2454] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6542b7192446f798840c60e86db31d in term 1.
I20260809 04:35:05.116258 21529 leader_election.cc:302] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a4d63d7d99429f994f1e9d0a95f639, 8f6542b7192446f798840c60e86db31d; no voters:
I20260809 04:35:05.116274 22023 raft_consensus.cc:2454] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6542b7192446f798840c60e86db31d in term 1.
I20260809 04:35:05.116475 22138 raft_consensus.cc:2781] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:05.116725 22138 raft_consensus.cc:686] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 1 LEADER]: Becoming Leader. State: Replica: 8f6542b7192446f798840c60e86db31d, State: Running, Role: LEADER
I20260809 04:35:05.117010 22138 consensus_queue.cc:237] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.119438 21407 catalog_manager.cc:5495] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f6542b7192446f798840c60e86db31d (127.19.92.1). New cstate: current_term: 1 leader_uuid: "8f6542b7192446f798840c60e86db31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } health_report { overall_health: UNKNOWN } } }
Aug 09 04:35:05 dist-test-slave-rmhg krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250103, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:05.181015 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.171148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45231 (local address 127.0.0.1:35496)
0809 04:35:05.171653 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:05.171667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:05.171783 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:05.172136 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:05.172138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:05.172662 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:05.172767 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.172774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.173846 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.173848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.174423 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.174431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.174991 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.174994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.175053 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.175717 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:05.175730 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:05.177438 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:05.179326 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:05.179336 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:05.179341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:05.179584 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:05.179884 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:05.179886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:05.179888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:05.179978 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:05.180447 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:05.180454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:05.180574 (+ 120us) client_negotiation.cc:769] Sending connection context
0809 04:35:05.180667 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 04:35:05.180809 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":168,"threads_started":1}
I20260809 04:35:05.181054 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.171249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.19.92.62:45231)
0809 04:35:05.171602 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.171611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.171750 (+ 139us) server_negotiation.cc:407] Connection header received
0809 04:35:05.171869 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.171873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.171925 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.172120 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:05.172882 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.173746 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.174544 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.174899 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.174989 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.177584 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:05.177588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:05.177590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:05.177616 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:05.179215 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:05.179702 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:05.179706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:05.179707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:05.179801 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:05.180056 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:05.180059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:05.180061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:05.180349 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:05.180450 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.180739 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.180873 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":169,"threads_started":1}
I20260809 04:35:05.185814 21407 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:35496:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 04:35:05.186570 21407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:05.191785 21593 tablet_service.cc:1467] Processing CreateTablet for tablet f574bf422d0945a4a20118b905996820 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=20ae0aa837d9474d802af547497ee60d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:05.191866 21796 tablet_service.cc:1467] Processing CreateTablet for tablet f574bf422d0945a4a20118b905996820 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=20ae0aa837d9474d802af547497ee60d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:05.192256 21796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f574bf422d0945a4a20118b905996820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:05.192267 21593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f574bf422d0945a4a20118b905996820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:05.195153 22131 tablet_bootstrap.cc:492] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: Bootstrap starting.
I20260809 04:35:05.195746 21993 tablet_service.cc:1467] Processing CreateTablet for tablet f574bf422d0945a4a20118b905996820 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=20ae0aa837d9474d802af547497ee60d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:05.196177 21993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f574bf422d0945a4a20118b905996820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:05.196759 22131 tablet_bootstrap.cc:654] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:05.198701 22131 tablet_bootstrap.cc:492] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: No bootstrap required, opened a new log
I20260809 04:35:05.198789 22131 ts_tablet_manager.cc:1397] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 04:35:05.199394 22132 tablet_bootstrap.cc:492] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: Bootstrap starting.
I20260809 04:35:05.199378 22131 raft_consensus.cc:348] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.199489 22131 raft_consensus.cc:374] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:05.199519 22131 raft_consensus.cc:729] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a4d63d7d99429f994f1e9d0a95f639, State: Initialized, Role: FOLLOWER
I20260809 04:35:05.199682 22131 consensus_queue.cc:260] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.199961 22141 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.200124 22141 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:05.200153 22130 tablet_bootstrap.cc:492] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: Bootstrap starting.
I20260809 04:35:05.200382 22132 tablet_bootstrap.cc:654] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:05.201025 22131 ts_tablet_manager.cc:1428] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 04:35:05.201071 22130 tablet_bootstrap.cc:654] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:05.202082 22132 tablet_bootstrap.cc:492] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: No bootstrap required, opened a new log
I20260809 04:35:05.202184 22132 ts_tablet_manager.cc:1397] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 04:35:05.202229 22130 tablet_bootstrap.cc:492] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: No bootstrap required, opened a new log
I20260809 04:35:05.202301 22130 ts_tablet_manager.cc:1397] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:05.202654 22132 raft_consensus.cc:348] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.202749 22132 raft_consensus.cc:374] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:05.202701 22130 raft_consensus.cc:348] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.202780 22132 raft_consensus.cc:729] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e151f99fd6304a89aa592e844dfba9e2, State: Initialized, Role: FOLLOWER
I20260809 04:35:05.202827 22130 raft_consensus.cc:374] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:05.202867 22130 raft_consensus.cc:729] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6542b7192446f798840c60e86db31d, State: Initialized, Role: FOLLOWER
I20260809 04:35:05.202925 22132 consensus_queue.cc:260] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.203018 22130 consensus_queue.cc:260] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.203202 22132 ts_tablet_manager.cc:1428] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:05.203187 22158 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.203285 22130 ts_tablet_manager.cc:1428] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:05.203353 22158 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:05.203464 22138 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.203586 22138 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:05.423496 22158 raft_consensus.cc:482] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:05.423647 22158 raft_consensus.cc:504] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.424059 22158 leader_election.cc:288] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f6542b7192446f798840c60e86db31d (127.19.92.1:45137), 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433)
I20260809 04:35:05.424595 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f574bf422d0945a4a20118b905996820" candidate_uuid: "e151f99fd6304a89aa592e844dfba9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6542b7192446f798840c60e86db31d" is_pre_election: true
I20260809 04:35:05.424636 21819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f574bf422d0945a4a20118b905996820" candidate_uuid: "e151f99fd6304a89aa592e844dfba9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a4d63d7d99429f994f1e9d0a95f639" is_pre_election: true
I20260809 04:35:05.424752 21619 raft_consensus.cc:2454] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e151f99fd6304a89aa592e844dfba9e2 in term 0.
I20260809 04:35:05.424791 21819 raft_consensus.cc:2454] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e151f99fd6304a89aa592e844dfba9e2 in term 0.
I20260809 04:35:05.425084 21927 leader_election.cc:302] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a4d63d7d99429f994f1e9d0a95f639, e151f99fd6304a89aa592e844dfba9e2; no voters:
I20260809 04:35:05.425282 22158 raft_consensus.cc:2781] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:05.425355 22158 raft_consensus.cc:482] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:05.425395 22158 raft_consensus.cc:3037] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:05.426389 22158 raft_consensus.cc:504] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.426716 22158 leader_election.cc:288] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [CANDIDATE]: Term 1 election: Requested vote from peers 8f6542b7192446f798840c60e86db31d (127.19.92.1:45137), 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433)
I20260809 04:35:05.427102 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f574bf422d0945a4a20118b905996820" candidate_uuid: "e151f99fd6304a89aa592e844dfba9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6542b7192446f798840c60e86db31d"
I20260809 04:35:05.427119 21819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f574bf422d0945a4a20118b905996820" candidate_uuid: "e151f99fd6304a89aa592e844dfba9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a4d63d7d99429f994f1e9d0a95f639"
I20260809 04:35:05.427227 21619 raft_consensus.cc:3037] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:05.427229 21819 raft_consensus.cc:3037] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:05.428130 21819 raft_consensus.cc:2454] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e151f99fd6304a89aa592e844dfba9e2 in term 1.
I20260809 04:35:05.428259 21619 raft_consensus.cc:2454] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e151f99fd6304a89aa592e844dfba9e2 in term 1.
I20260809 04:35:05.428417 21927 leader_election.cc:302] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a4d63d7d99429f994f1e9d0a95f639, e151f99fd6304a89aa592e844dfba9e2; no voters:
I20260809 04:35:05.428587 22158 raft_consensus.cc:2781] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:05.428678 22158 raft_consensus.cc:686] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [term 1 LEADER]: Becoming Leader. State: Replica: e151f99fd6304a89aa592e844dfba9e2, State: Running, Role: LEADER
I20260809 04:35:05.428825 22158 consensus_queue.cc:237] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } }
I20260809 04:35:05.429585 22143 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: TxnStatusTablet state changed. Reason: New leader e151f99fd6304a89aa592e844dfba9e2. Latest consensus state: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.429705 22143 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:05.429997 22195 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 04:35:05.430210 22195 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 04:35:05.430711 21407 catalog_manager.cc:5495] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e151f99fd6304a89aa592e844dfba9e2 (127.19.92.3). New cstate: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } health_report { overall_health: HEALTHY } } }
I20260809 04:35:05.444617 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.441221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:40601 (local address 127.0.0.1:45092)
0809 04:35:05.441350 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:05.441367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:05.441461 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:05.441695 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:05.441697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:05.441705 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:05.441795 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.441800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.442760 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.442763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.443381 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.443388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.443901 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.443904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.443977 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.444026 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:05.444322 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:05.444341 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:35:05.444423 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 04:35:05.444500 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 04:35:05.444664 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.441305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.19.92.3:40601)
0809 04:35:05.441443 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.441447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.441462 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:35:05.441552 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.441555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.441606 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.441689 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:05.441875 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.442660 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.443475 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.443815 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.443911 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.444133 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:05.444216 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:05.444283 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.444464 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.444524 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:35:05.447557 21819 raft_consensus.cc:1261] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Refusing update from remote peer e151f99fd6304a89aa592e844dfba9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:05.447589 21619 raft_consensus.cc:1261] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Refusing update from remote peer e151f99fd6304a89aa592e844dfba9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:05.448002 22143 consensus_queue.cc:1035] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:05.448185 22158 consensus_queue.cc:1035] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:05.449908 22141 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: TxnStatusTablet state changed. Reason: New leader e151f99fd6304a89aa592e844dfba9e2. Latest consensus state: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.450042 22141 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:05.450492 22138 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: TxnStatusTablet state changed. Reason: New leader e151f99fd6304a89aa592e844dfba9e2. Latest consensus state: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.450608 22138 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:05.453543 22140 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.453673 22140 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:05.453797 22140 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.453867 22140 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P e151f99fd6304a89aa592e844dfba9e2: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:05.456570 22138 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.456691 22138 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P 8f6542b7192446f798840c60e86db31d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:05.456749 22141 tablet_replica.cc:439] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e151f99fd6304a89aa592e844dfba9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6542b7192446f798840c60e86db31d" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45137 } } peers { permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 } } peers { permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 } } }
I20260809 04:35:05.456880 22141 tablet_replica.cc:442] T f574bf422d0945a4a20118b905996820 P 38a4d63d7d99429f994f1e9d0a95f639: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:05.465735 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.461307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.19.92.1:45137)
0809 04:35:05.461906 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.461914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.461927 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:05.461999 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.462002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.462054 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.462262 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:05.462533 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.463243 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.464046 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.464336 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.464463 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.464740 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:05.465268 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:05.465376 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.465450 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.465507 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":348,"threads_started":1}
I20260809 04:35:05.478677 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.474408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:45177 (local address 127.19.92.3:40601)
0809 04:35:05.474573 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.474578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.474830 (+ 252us) server_negotiation.cc:407] Connection header received
0809 04:35:05.474960 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.474962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.475010 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.475071 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:05.475670 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.476176 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.477592 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.477826 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.477902 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.478290 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:05.478362 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:05.478407 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.478505 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.478557 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:35:05.478825 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.474273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:40601 (local address 127.19.92.1:45177)
0809 04:35:05.474743 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:05.474758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:05.474872 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:05.475097 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:05.475100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:05.475108 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:05.475508 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.475517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.476273 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.476277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.477457 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.477470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.477921 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.477925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.478096 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.478160 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:05.478431 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:05.478437 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:35:05.478499 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 04:35:05.478582 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":150,"threads_started":1}
I20260809 04:35:05.486543 21819 raft_consensus.cc:1261] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Refusing update from remote peer 8f6542b7192446f798840c60e86db31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:05.486790 22023 raft_consensus.cc:1261] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Refusing update from remote peer 8f6542b7192446f798840c60e86db31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:05.487061 22175 consensus_queue.cc:1035] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a4d63d7d99429f994f1e9d0a95f639" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:05.487299 22138 consensus_queue.cc:1035] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e151f99fd6304a89aa592e844dfba9e2" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:05.511780 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.507742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:51043 (local address 127.19.92.1:45137)
0809 04:35:05.507860 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.507865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.508345 (+ 480us) server_negotiation.cc:407] Connection header received
0809 04:35:05.508551 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.508554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.508614 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.508658 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:05.509128 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.509803 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.510777 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.511008 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.511096 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.511367 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:05.511439 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:05.511495 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.511591 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.511643 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:05.512097 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.507645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45137 (local address 127.19.92.3:51043)
0809 04:35:05.508137 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:05.508154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:05.508410 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:05.508799 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:05.508802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:05.508810 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:05.508913 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.508918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.509927 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.509930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.510664 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:05.510675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.511103 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.511107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:05.511192 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.511245 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:05.511517 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:05.511524 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:35:05.511582 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 04:35:05.511714 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":160,"threads_started":1}
I20260809 04:35:05.542057 21407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:05.542229 21407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:05.544513 21407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e11bdc5e6257474eaffe225f2d639ea4: Sending DeleteTablet for 3 replicas of tablet ff0c7b1640484c22a89b3faf67dd4f63
I20260809 04:35:05.545190 21796 tablet_service.cc:1514] Processing DeleteTablet for tablet ff0c7b1640484c22a89b3faf67dd4f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:05 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:54148
I20260809 04:35:05.545614 21593 tablet_service.cc:1514] Processing DeleteTablet for tablet ff0c7b1640484c22a89b3faf67dd4f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:05 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:35406
I20260809 04:35:05.546739 21993 tablet_service.cc:1514] Processing DeleteTablet for tablet ff0c7b1640484c22a89b3faf67dd4f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:05 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:45080
I20260809 04:35:05.549388 22216 tablet_replica.cc:330] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: stopping tablet replica
I20260809 04:35:05.549580 22216 raft_consensus.cc:2229] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:05.550067 22216 raft_consensus.cc:2258] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:05.550920 22216 ts_tablet_manager.cc:1905] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:05.554837 22216 ts_tablet_manager.cc:1918] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:05.554920 22216 log.cc:1196] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/ff0c7b1640484c22a89b3faf67dd4f63
I20260809 04:35:05.555194 22216 ts_tablet_manager.cc:1939] T ff0c7b1640484c22a89b3faf67dd4f63 P 8f6542b7192446f798840c60e86db31d: Deleting consensus metadata
I20260809 04:35:05.555974 21392 catalog_manager.cc:4841] TS 8f6542b7192446f798840c60e86db31d (127.19.92.1:45137): tablet ff0c7b1640484c22a89b3faf67dd4f63 (table test-table [id=8456c8b6959b4c148f45fd9d72eacb61]) successfully deleted
Aug 09 04:35:05 dist-test-slave-rmhg krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:35:05.563555 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.547077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.19.92.1:45137)
0809 04:35:05.547200 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.547204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.550628 (+ 3424us) server_negotiation.cc:407] Connection header received
0809 04:35:05.550805 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.550808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.550853 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.550910 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:05.552278 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.553082 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.554621 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.554983 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.555051 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.559731 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:05.559750 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:05.559767 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:05.559796 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:05.561657 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:05.562072 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:05.562077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:05.562079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:05.562155 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:05.562410 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:05.562412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:05.562414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:05.562664 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:05.562996 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.563153 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.563270 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260809 04:35:05.564273 21679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35428
I20260809 04:35:05.564770 22219 tablet_replica.cc:330] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: stopping tablet replica
I20260809 04:35:05.564917 22219 raft_consensus.cc:2229] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:05.565058 22219 raft_consensus.cc:2258] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:05.565888 22219 ts_tablet_manager.cc:1905] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:05.569125 22219 ts_tablet_manager.cc:1918] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:05.569242 22219 log.cc:1196] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/ff0c7b1640484c22a89b3faf67dd4f63
I20260809 04:35:05.569356 22215 tablet_replica.cc:330] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: stopping tablet replica
I20260809 04:35:05.569519 22219 ts_tablet_manager.cc:1939] T ff0c7b1640484c22a89b3faf67dd4f63 P e151f99fd6304a89aa592e844dfba9e2: Deleting consensus metadata
I20260809 04:35:05.569510 22215 raft_consensus.cc:2229] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:05.569973 22215 raft_consensus.cc:2258] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:05.570314 21392 catalog_manager.cc:4841] TS e151f99fd6304a89aa592e844dfba9e2 (127.19.92.3:40601): tablet ff0c7b1640484c22a89b3faf67dd4f63 (table test-table [id=8456c8b6959b4c148f45fd9d72eacb61]) successfully deleted
I20260809 04:35:05.570792 22215 ts_tablet_manager.cc:1905] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:05.574246 22215 ts_tablet_manager.cc:1918] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:05.574338 22215 log.cc:1196] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/ff0c7b1640484c22a89b3faf67dd4f63
I20260809 04:35:05.574631 22215 ts_tablet_manager.cc:1939] T ff0c7b1640484c22a89b3faf67dd4f63 P 38a4d63d7d99429f994f1e9d0a95f639: Deleting consensus metadata
I20260809 04:35:05.574976 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:05.566105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.19.92.62:45231)
0809 04:35:05.566227 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:05.566231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:05.566519 (+ 288us) server_negotiation.cc:407] Connection header received
0809 04:35:05.566628 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:05.566632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:05.566688 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:05.566749 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:05.567377 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.567910 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.568714 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:05.568964 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:05.569056 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:05.571059 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:05.571063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:05.571066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:05.571096 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:05.572790 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:05.573400 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:05.573404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:05.573406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:05.573541 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:05.573855 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:05.573858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:05.573860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:05.574097 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:05.574207 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:05.574643 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 04:35:05.574802 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 04:35:05.575385 21392 catalog_manager.cc:4841] TS 38a4d63d7d99429f994f1e9d0a95f639 (127.19.92.2:44433): tablet ff0c7b1640484c22a89b3faf67dd4f63 (table test-table [id=8456c8b6959b4c148f45fd9d72eacb61]) successfully deleted
W20260809 04:35:05.576287 21407 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35500
I20260809 04:35:05.578156 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21505
I20260809 04:35:05.608366 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21703
I20260809 04:35:05.620115 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21903
I20260809 04:35:05.635239 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21366
2026-08-09T04:35:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3324 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:05 dist-test-slave-rmhg krb5kdc[22225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:05 dist-test-slave-rmhg krb5kdc[22225](info): set up 2 sockets
Aug 09 04:35:05 dist-test-slave-rmhg krb5kdc[22225](info): commencing operation
krb5kdc: starting...
W20260809 04:35:06.769423 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:06 dist-test-slave-rmhg krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:06.899607 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:34885
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:07.019917 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:07.020200 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:07.020241 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:07.024272 22245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:07.024336 22245 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 04:35:07.024355 22245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:07.024374 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:07.024403 22245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:07.024497 22245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:07.029878 22245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:34885
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:07.030980 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:07.032014 22245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:07.040369 22252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:07.040335 22253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:07.040336 22258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:07.040433 22245 server_base.cc:1034] running on GCE node
I20260809 04:35:07.041229 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:07.042268 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:07.043426 22245 hybrid_clock.cc:648] HybridClock initialized: now 1786250107043406 us; error 44 us; skew 500 ppm
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:07.048985 22245 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:07.049938 22245 webserver.cc:460] Webserver started at http://127.19.92.62:39641/ using document root <none> and password file <none>
I20260809 04:35:07.050453 22245 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:07.050510 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:07.050688 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:07.052758 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "f07fbaf1e6784cada7a281c129a5b314"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "94921c212b709316539f74620d8aadf4"
server_key_iv: "477b0d6024ba58c29511f46cb92aece9"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.053381 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "f07fbaf1e6784cada7a281c129a5b314"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "94921c212b709316539f74620d8aadf4"
server_key_iv: "477b0d6024ba58c29511f46cb92aece9"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.057451 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:35:07.059909 22265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:07.061251 22245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:07.061375 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "f07fbaf1e6784cada7a281c129a5b314"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "94921c212b709316539f74620d8aadf4"
server_key_iv: "477b0d6024ba58c29511f46cb92aece9"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.061512 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:07.077814 22245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:07.080675 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:07.080834 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:07.088663 22245 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:34885
I20260809 04:35:07.088735 22351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:34885 every 8 connection(s)
I20260809 04:35:07.089668 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:07.092537 22352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:07.097193 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 22245
I20260809 04:35:07.097319 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:07.097604 19824 external_mini_cluster.cc:1428] Setting key beb8360b015ab93c79b55e4827a087de
I20260809 04:35:07.098948 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: Bootstrap starting.
I20260809 04:35:07.101475 22352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:07.102228 22352 log.cc:824] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:07.104279 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: No bootstrap required, opened a new log
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:07.108251 22352 raft_consensus.cc:348] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07fbaf1e6784cada7a281c129a5b314" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34885 } }
I20260809 04:35:07.108520 22352 raft_consensus.cc:374] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:07.108588 22352 raft_consensus.cc:729] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07fbaf1e6784cada7a281c129a5b314, State: Initialized, Role: FOLLOWER
I20260809 04:35:07.109031 22352 consensus_queue.cc:260] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07fbaf1e6784cada7a281c129a5b314" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34885 } }
I20260809 04:35:07.109184 22352 raft_consensus.cc:388] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:07.109257 22352 raft_consensus.cc:482] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:07.109373 22352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:07.110427 22352 raft_consensus.cc:504] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07fbaf1e6784cada7a281c129a5b314" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34885 } }
I20260809 04:35:07.110805 22352 leader_election.cc:302] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f07fbaf1e6784cada7a281c129a5b314; no voters:
I20260809 04:35:07.111223 22352 leader_election.cc:288] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:07.111310 22359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:07.111515 22359 raft_consensus.cc:686] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [term 1 LEADER]: Becoming Leader. State: Replica: f07fbaf1e6784cada7a281c129a5b314, State: Running, Role: LEADER
I20260809 04:35:07.111826 22359 consensus_queue.cc:237] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07fbaf1e6784cada7a281c129a5b314" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34885 } }
I20260809 04:35:07.112190 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.099187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.19.92.62:34885)
0809 04:35:07.099677 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.099694 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.099732 (+ 38us) server_negotiation.cc:407] Connection header received
0809 04:35:07.100407 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.100425 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.100769 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.101172 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:07.101936 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.103168 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.104021 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.104557 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.104719 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.107377 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:07.107382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:07.107396 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:07.107425 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:07.109096 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.109640 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.109645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.109649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.109763 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.110037 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.110039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.110040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.110433 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:07.110635 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.110889 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.111218 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":152,"threads_started":1}
I20260809 04:35:07.112568 22352 sys_catalog.cc:564] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:07.113049 22359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f07fbaf1e6784cada7a281c129a5b314. Latest consensus state: current_term: 1 leader_uuid: "f07fbaf1e6784cada7a281c129a5b314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07fbaf1e6784cada7a281c129a5b314" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34885 } } }
I20260809 04:35:07.113204 22359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:07.113498 22360 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f07fbaf1e6784cada7a281c129a5b314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07fbaf1e6784cada7a281c129a5b314" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34885 } } }
I20260809 04:35:07.114538 22360 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314 [sys.catalog]: This master's current role is: LEADER
W20260809 04:35:07.117479 22376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:35:07.117548 22376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:35:07.117640 22375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:07.118335 22375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:07.124104 22375 catalog_manager.cc:1347] Generated new cluster ID: ccf6032b29484f418f8b2328f8732f21
I20260809 04:35:07.124176 22375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:07.148769 22375 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:07.149788 22375 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:07.161831 22375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: Generated new TSK 0
I20260809 04:35:07.162531 22375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:07.202203 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34885
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:07.336474 22385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:07.336779 22385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:07.336829 22385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:07.340837 22385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:07.340898 22385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:07.340937 22385 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 04:35:07.340950 22385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:07.340982 22385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:07.346310 22385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:07.347402 22385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:07.348410 22385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:07.357996 22393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:07.357996 22391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:07.358027 22395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:07.358093 22385 server_base.cc:1034] running on GCE node
I20260809 04:35:07.358626 22385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:07.359308 22385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:07.360464 22385 hybrid_clock.cc:648] HybridClock initialized: now 1786250107360438 us; error 41 us; skew 500 ppm
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:07.363221 22385 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:07.364064 22385 webserver.cc:460] Webserver started at http://127.19.92.1:44983/ using document root <none> and password file <none>
I20260809 04:35:07.364645 22385 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:07.364718 22385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:07.364964 22385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:07.367111 22385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "86acf733d80244159f20abf759fb3603"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "af8a88cde1827452b8dd9998aaa7977c"
server_key_iv: "d2eda5181ce59838e30d076e2f0eabba"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.367667 22385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "86acf733d80244159f20abf759fb3603"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "af8a88cde1827452b8dd9998aaa7977c"
server_key_iv: "d2eda5181ce59838e30d076e2f0eabba"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.371483 22385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:35:07.374155 22405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:07.375629 22385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:07.375738 22385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "86acf733d80244159f20abf759fb3603"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "af8a88cde1827452b8dd9998aaa7977c"
server_key_iv: "d2eda5181ce59838e30d076e2f0eabba"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.375842 22385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:07.409358 22385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:07.412986 22385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:07.413206 22385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:07.414908 22385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:07.414975 22385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:07.415019 22385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:07.415043 22385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:07.425355 22385 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:38597
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:07.425414 22564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:38597 every 8 connection(s)
I20260809 04:35:07.426359 22385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:07.429421 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.415849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:55475 (local address 127.19.92.62:34885)
0809 04:35:07.415976 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.415980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.417190 (+ 1210us) server_negotiation.cc:407] Connection header received
0809 04:35:07.418286 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.418291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.418356 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.418472 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:07.420092 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.420934 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.422615 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.422975 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.423103 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.426046 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:07.426050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:07.426053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:07.426085 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:07.427706 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.428156 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.428159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.428160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.428236 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.428494 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.428496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.428497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.428664 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:07.428735 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.429192 (+ 457us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.429297 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:35:07.430142 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 22385
I20260809 04:35:07.430269 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:07.430388 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.416497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34885 (local address 127.19.92.1:55475)
0809 04:35:07.416990 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.417052 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.418037 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.418666 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.418676 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.419087 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:07.419899 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.419921 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.421080 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.421084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.422474 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.422485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.423170 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.423175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.423379 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.424048 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:07.424071 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:07.425869 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:07.427806 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.427811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.427822 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.428072 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.428299 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.428301 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.428303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.428419 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.428734 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:07.428738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:07.428975 (+ 237us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.429117 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.429422 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
I20260809 04:35:07.430601 19824 external_mini_cluster.cc:1428] Setting key 85a0a2e7cba85e7892f7b3b2808dbd56
I20260809 04:35:07.432014 22566 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34885
I20260809 04:35:07.432153 22566 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:07.432739 22566 heartbeater.cc:511] Master 127.19.92.62:34885 requested a full tablet report, sending...
I20260809 04:35:07.434345 22289 ts_manager.cc:194] Registered new tserver with Master: 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597)
I20260809 04:35:07.435686 22289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:55475
I20260809 04:35:07.441532 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.434335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:52881 (local address 127.19.92.62:34885)
0809 04:35:07.434441 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.434446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.434671 (+ 225us) server_negotiation.cc:407] Connection header received
0809 04:35:07.434780 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.434783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.434823 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.434893 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:07.435352 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.436051 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.437177 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.437491 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.437597 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.438762 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:07.438765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:07.438767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:07.438789 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:07.440019 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.440458 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.440460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.440461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.440527 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.440771 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.440772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.440773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.440919 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:07.440987 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.441293 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.441383 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 04:35:07.441601 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.434246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34885 (local address 127.19.92.1:52881)
0809 04:35:07.434585 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.434599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.434693 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.434916 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.434918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.435122 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:07.435241 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.435246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.436194 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.436197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.437029 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.437041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.437621 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.437625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.437720 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.438174 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:07.438187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:07.438622 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:07.440122 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.440126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.440128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.440371 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.440602 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.440605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.440607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.440695 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.441003 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:07.441007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:07.441118 (+ 111us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.441253 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.441397 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:07.460450 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34885
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:07.574318 22582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:07.574633 22582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:07.574714 22582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:07.578938 22582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:07.579011 22582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:07.579070 22582 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 04:35:07.579095 22582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:07.579144 22582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:07.584767 22582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:07.585916 22582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:07.586985 22582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:07.596897 22587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:07.596964 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:07.596930 22582 server_base.cc:1034] running on GCE node
W20260809 04:35:07.596897 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:07.597585 22582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:07.598253 22582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:07.599421 22582 hybrid_clock.cc:648] HybridClock initialized: now 1786250107599401 us; error 34 us; skew 500 ppm
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:07.602257 22582 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:07.603240 22582 webserver.cc:460] Webserver started at http://127.19.92.2:44981/ using document root <none> and password file <none>
I20260809 04:35:07.603820 22582 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:07.603890 22582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:07.604094 22582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:07.606266 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "ed4102b376adb69e4f20751db5d2b357"
server_key_iv: "9761cc250c7e8cafac1ddb56f178329c"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.606828 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "ed4102b376adb69e4f20751db5d2b357"
server_key_iv: "9761cc250c7e8cafac1ddb56f178329c"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.610782 22582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:35:07.613425 22605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:07.614892 22582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:07.615020 22582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "ed4102b376adb69e4f20751db5d2b357"
server_key_iv: "9761cc250c7e8cafac1ddb56f178329c"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.615137 22582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:07.636297 22582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:07.639225 22582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:07.639401 22582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:07.640888 22582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:07.640940 22582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:07.640995 22582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:07.641023 22582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:07.650940 22582 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:37613
I20260809 04:35:07.651062 22769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:37613 every 8 connection(s)
I20260809 04:35:07.651857 22582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:07.654919 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.641719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:48957 (local address 127.19.92.62:34885)
0809 04:35:07.641876 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.641881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.642757 (+ 876us) server_negotiation.cc:407] Connection header received
0809 04:35:07.643843 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.643847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.643908 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.643991 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:07.645675 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.646498 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.648231 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.648453 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.648538 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.651505 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:07.651510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:07.651512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:07.651541 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:07.652996 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.653553 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.653556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.653557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.653644 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.653942 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.653944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.653946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.654130 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:07.654201 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.654673 (+ 472us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.654792 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:35:07.655797 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.642130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34885 (local address 127.19.92.2:48957)
0809 04:35:07.642587 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.642637 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.643606 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.644213 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.644222 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.644713 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:07.645478 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.645501 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.646632 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.646636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.648109 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.648121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.648609 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.648613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.648824 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.649569 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:07.649594 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:07.651336 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:07.653103 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.653109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.653124 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.653451 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.653720 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.653722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.653723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.653866 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.654203 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:07.654208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:07.654455 (+ 247us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.654616 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.654898 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260809 04:35:07.657018 22770 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34885
I20260809 04:35:07.657195 22770 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:07.657740 22770 heartbeater.cc:511] Master 127.19.92.62:34885 requested a full tablet report, sending...
I20260809 04:35:07.658339 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 22582
I20260809 04:35:07.658447 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:07.658723 19824 external_mini_cluster.cc:1428] Setting key c76b28995c879cb4650a5f379ff8997d
I20260809 04:35:07.658874 22287 ts_manager.cc:194] Registered new tserver with Master: bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613)
I20260809 04:35:07.659618 22287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:48957
I20260809 04:35:07.667737 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.659329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:36481 (local address 127.19.92.62:34885)
0809 04:35:07.659437 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.659441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.661205 (+ 1764us) server_negotiation.cc:407] Connection header received
0809 04:35:07.661253 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.661255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.661301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.661377 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:07.661876 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.662438 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.663268 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.663511 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.663597 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.664742 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:07.664745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:07.664747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:07.664770 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:07.666212 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.666664 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.666666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.666667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.666737 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.666984 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.666986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.666987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.667194 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:07.667268 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.667528 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.667619 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:35:07.667909 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.659265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34885 (local address 127.19.92.2:36481)
0809 04:35:07.659866 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.659881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.659974 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.661411 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.661414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.661658 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:07.661774 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.661780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.662561 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.662564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.663162 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.663170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.663610 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.663613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.663701 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.664144 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:07.664159 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:07.664579 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:07.666312 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.666316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.666318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.666575 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.666809 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.666812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.666814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.666907 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.667279 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:07.667282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:07.667382 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.667501 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.667640 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:07.686470 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34885
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:07.800508 22783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:07.800756 22783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:07.800801 22783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:07.804772 22783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:07.804832 22783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:07.804870 22783 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 04:35:07.804883 22783 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:07.804916 22783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:07.809979 22783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:07.810971 22783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:07.811977 22783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:07.822180 22794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:07.822183 22792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:07.822181 22796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:07.822443 22783 server_base.cc:1034] running on GCE node
I20260809 04:35:07.822860 22783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:07.823500 22783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:07.824662 22783 hybrid_clock.cc:648] HybridClock initialized: now 1786250107824644 us; error 32 us; skew 500 ppm
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:07.827382 22783 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:07.828246 22783 webserver.cc:460] Webserver started at http://127.19.92.3:36035/ using document root <none> and password file <none>
I20260809 04:35:07.828791 22783 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:07.828856 22783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:07.829047 22783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:07.831141 22783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "8875775338674f0d82511974d1dbb1ca"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "0eedd4db64abb37e3c57a999dff85f87"
server_key_iv: "1a7e6ddcde0540ebe74f6719b4d616bc"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.831713 22783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "8875775338674f0d82511974d1dbb1ca"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "0eedd4db64abb37e3c57a999dff85f87"
server_key_iv: "1a7e6ddcde0540ebe74f6719b4d616bc"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.835793 22783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 04:35:07.838346 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:07.839625 22783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:07.839737 22783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "8875775338674f0d82511974d1dbb1ca"
format_stamp: "Formatted at 2026-08-09 04:35:07 on dist-test-slave-rmhg"
server_key: "0eedd4db64abb37e3c57a999dff85f87"
server_key_iv: "1a7e6ddcde0540ebe74f6719b4d616bc"
server_key_version: "encryptionkey@0"
I20260809 04:35:07.839831 22783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:07.867625 22783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:07.870638 22783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:07.870815 22783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:07.872323 22783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:07.872391 22783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:07.872449 22783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:07.872475 22783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:07.883370 22783 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:34649
I20260809 04:35:07.883392 22984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:34649 every 8 connection(s)
I20260809 04:35:07.884377 22783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:07.886924 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.873277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:52067 (local address 127.19.92.62:34885)
0809 04:35:07.873394 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.873397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.874517 (+ 1120us) server_negotiation.cc:407] Connection header received
0809 04:35:07.875561 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.875564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.875627 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.875707 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:07.877392 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.878229 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.879949 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.880287 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.880378 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.883129 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:07.883133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:07.883136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:07.883162 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:07.884755 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.885427 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.885432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.885434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.885537 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.885871 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.885874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.885876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.886062 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:07.886148 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.886634 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.886759 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:35:07.887640 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.873851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34885 (local address 127.19.92.3:52067)
0809 04:35:07.874339 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.874384 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.875345 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.875950 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.875960 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.876477 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:07.877226 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.877250 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.878370 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.878374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.879825 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.879838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.880455 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.880460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.880659 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.881302 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:07.881322 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:07.882955 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:07.884871 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.884877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.884928 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.885309 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.885630 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.885633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.885635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.885771 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.886165 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:07.886172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:07.886397 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.886558 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.886868 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":129,"threads_started":1}
I20260809 04:35:07.888760 22985 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34885
I20260809 04:35:07.888904 22985 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:07.889468 22985 heartbeater.cc:511] Master 127.19.92.62:34885 requested a full tablet report, sending...
I20260809 04:35:07.890540 22289 ts_manager.cc:194] Registered new tserver with Master: 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649)
I20260809 04:35:07.891253 22289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:52067
I20260809 04:35:07.893675 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 22783
I20260809 04:35:07.893796 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:07.894074 19824 external_mini_cluster.cc:1428] Setting key 24c7fef14e819954167d83b3f5d275ad
I20260809 04:35:07.896137 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:35:07.898314 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.891008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34885 (local address 127.19.92.3:45925)
0809 04:35:07.891256 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.891267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.891354 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.891593 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.891596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.891805 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:07.891930 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.891937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.892904 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.892907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.893602 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.893613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.894105 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.894109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.894207 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.894675 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:07.894691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:07.895095 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:07.896921 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.896923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.896925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.897141 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.897458 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:07.897460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:07.897461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:07.897538 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:07.897890 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:07.897892 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:07.897969 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.898054 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.898165 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260809 04:35:07.898381 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.891072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:45925 (local address 127.19.92.62:34885)
0809 04:35:07.891178 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.891182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.891326 (+ 144us) server_negotiation.cc:407] Connection header received
0809 04:35:07.891439 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.891443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.891486 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.891562 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:07.892049 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.892776 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.893716 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.893997 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.894088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.895215 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:07.895219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:07.895222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:07.895250 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:07.896814 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.897269 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.897273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.897274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.897367 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.897608 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.897611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.897613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.897815 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:07.897908 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.898121 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.898238 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:35:07 dist-test-slave-rmhg krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:07.923314 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.915540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.19.92.62:34885)
0809 04:35:07.915667 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.915671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.915685 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:07.915726 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.915728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.915770 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.915839 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:07.916523 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.917048 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.917931 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.918148 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.918220 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.920577 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:07.920581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:07.920583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:07.920608 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:07.921921 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.922300 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.922303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.922304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.922374 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:07.922622 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:07.922626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:07.922627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:07.922815 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:07.922919 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.923089 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.923184 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:07.926180 22289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:07.929076 22289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:07.943920 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.937879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:37613 (local address 127.0.0.1:53400)
0809 04:35:07.938307 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.938362 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.938501 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.939583 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.939589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.939615 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:07.939741 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.939747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.941338 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.941341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.942471 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.942479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.943554 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.943557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.943639 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.943659 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.943716 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.943768 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260809 04:35:07.944222 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.938079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.19.92.2:37613)
0809 04:35:07.939071 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.939084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.939106 (+ 22us) server_negotiation.cc:407] Connection header received
0809 04:35:07.939177 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.939182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.939397 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.939893 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:07.940028 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.941227 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.942582 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.943451 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.943561 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.943747 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.943804 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.943896 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":154,"threads_started":1}
I20260809 04:35:07.944243 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.938645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38597 (local address 127.0.0.1:46266)
0809 04:35:07.938901 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.938913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.938995 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.939954 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.939956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.939974 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:07.940092 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.940098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.941424 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.941428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.942671 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.942682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.943853 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.943857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.943954 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.943979 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.944046 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.944123 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":177,"thread_start_us":110,"threads_started":1}
I20260809 04:35:07.944242 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.938740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.19.92.3:34649)
0809 04:35:07.939345 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.939357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.939381 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:35:07.939460 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.939466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.939652 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.939803 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:07.940019 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.940828 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.942399 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.943141 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.943283 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.943464 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.943881 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.944101 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":118,"threads_started":1}
I20260809 04:35:07.944640 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.939069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.19.92.1:38597)
0809 04:35:07.939419 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:07.939424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:07.939441 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:35:07.939501 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:07.939505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:07.939652 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:07.939804 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:07.940217 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.941318 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.942800 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.943585 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.943676 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.943790 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:07.944397 (+ 607us) server_negotiation.cc:299] Negotiation successful
0809 04:35:07.944516 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":73,"threads_started":1}
I20260809 04:35:07.943807 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:07.938197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:34649 (local address 127.0.0.1:37666)
0809 04:35:07.938738 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:07.938748 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:07.938824 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:07.939783 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:07.939786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:07.939803 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:07.939900 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.939905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.940946 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.940949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.942285 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:07.942297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:07.943249 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:07.943252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:07.943370 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:07.943408 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 04:35:07.943529 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 04:35:07.943646 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":115,"threads_started":1}
I20260809 04:35:07.947268 22675 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8ddf4243c9425c94e65f1b51ee6542 (DEFAULT_TABLE table=test-table [id=a8765fc7762b4220ad4600e94df96b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:07.947430 22883 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8ddf4243c9425c94e65f1b51ee6542 (DEFAULT_TABLE table=test-table [id=a8765fc7762b4220ad4600e94df96b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:07.947428 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8ddf4243c9425c94e65f1b51ee6542 (DEFAULT_TABLE table=test-table [id=a8765fc7762b4220ad4600e94df96b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:07.948240 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ddf4243c9425c94e65f1b51ee6542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:07.948462 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ddf4243c9425c94e65f1b51ee6542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:07.948455 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ddf4243c9425c94e65f1b51ee6542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:07.955152 23016 tablet_bootstrap.cc:492] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Bootstrap starting.
I20260809 04:35:07.956014 23017 tablet_bootstrap.cc:492] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Bootstrap starting.
I20260809 04:35:07.956486 23019 tablet_bootstrap.cc:492] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Bootstrap starting.
I20260809 04:35:07.957332 23016 tablet_bootstrap.cc:654] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:07.958098 23016 log.cc:824] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:07.958216 23017 tablet_bootstrap.cc:654] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:07.958489 23019 tablet_bootstrap.cc:654] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:07.959053 23017 log.cc:824] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:07.959256 23019 log.cc:824] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:07.960220 23016 tablet_bootstrap.cc:492] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: No bootstrap required, opened a new log
I20260809 04:35:07.960412 23016 ts_tablet_manager.cc:1397] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:35:07.960803 23017 tablet_bootstrap.cc:492] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: No bootstrap required, opened a new log
I20260809 04:35:07.960973 23017 ts_tablet_manager.cc:1397] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 04:35:07.961192 23019 tablet_bootstrap.cc:492] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: No bootstrap required, opened a new log
I20260809 04:35:07.961365 23019 ts_tablet_manager.cc:1397] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:35:07.964309 23017 raft_consensus.cc:348] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:07.964308 23016 raft_consensus.cc:348] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:07.964545 23017 raft_consensus.cc:374] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:07.964560 23016 raft_consensus.cc:374] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:07.964598 23017 raft_consensus.cc:729] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86acf733d80244159f20abf759fb3603, State: Initialized, Role: FOLLOWER
I20260809 04:35:07.964607 23016 raft_consensus.cc:729] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5f1a218f4a4840b7fb5616d3b2ac35, State: Initialized, Role: FOLLOWER
I20260809 04:35:07.965008 23017 consensus_queue.cc:260] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:07.965088 23016 consensus_queue.cc:260] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:07.965098 23019 raft_consensus.cc:348] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:07.965425 23019 raft_consensus.cc:374] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:07.965545 23019 raft_consensus.cc:729] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8875775338674f0d82511974d1dbb1ca, State: Initialized, Role: FOLLOWER
I20260809 04:35:07.965793 22566 heartbeater.cc:503] Master 127.19.92.62:34885 was elected leader, sending a full tablet report...
I20260809 04:35:07.965958 23017 ts_tablet_manager.cc:1428] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260809 04:35:07.966059 23016 ts_tablet_manager.cc:1428] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:35:07.966379 22770 heartbeater.cc:503] Master 127.19.92.62:34885 was elected leader, sending a full tablet report...
I20260809 04:35:07.966713 23019 consensus_queue.cc:260] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:07.967593 23019 ts_tablet_manager.cc:1428] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:35:07.967922 22985 heartbeater.cc:503] Master 127.19.92.62:34885 was elected leader, sending a full tablet report...
W20260809 04:35:08.135820 22986 tablet.cc:2369] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:35:08.153008 22773 tablet.cc:2369] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:35:08.177724 22567 tablet.cc:2369] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:35:08.365167 23026 raft_consensus.cc:482] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:08.365437 23026 raft_consensus.cc:504] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.366586 23026 leader_election.cc:288] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597), 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649)
I20260809 04:35:08.370543 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.367285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:60029 (local address 127.19.92.3:34649)
0809 04:35:08.367430 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.367434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.367448 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:08.367507 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.367509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.367549 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.367615 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:08.367891 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.368414 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.369597 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.369998 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.370094 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.370163 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.370333 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.370405 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:35:08.370556 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.366910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:34649 (local address 127.19.92.2:60029)
0809 04:35:08.367218 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.367234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.367349 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.367638 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.367642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.367659 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:08.367785 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.367790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.368521 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.368525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.369473 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.369484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.370118 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.370121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.370219 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.370240 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.370305 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.370388 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":42,"thread_start_us":101,"threads_started":1}
I20260809 04:35:08.370553 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.367023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:56309 (local address 127.19.92.1:38597)
0809 04:35:08.367219 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.367224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.367385 (+ 161us) server_negotiation.cc:407] Connection header received
0809 04:35:08.367580 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.367582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.367628 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.367691 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:08.367953 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.368473 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.369598 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.370027 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.370124 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.370185 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.370351 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.370432 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 04:35:08.370568 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.366910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38597 (local address 127.19.92.2:56309)
0809 04:35:08.367292 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.367306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.367463 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.367717 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.367720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.367736 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:08.367847 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.367852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.368582 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.368586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.369482 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.369492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.370144 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.370148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.370245 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.370263 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.370326 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.370401 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":34,"thread_start_us":74,"threads_started":1}
I20260809 04:35:08.371244 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8ddf4243c9425c94e65f1b51ee6542" candidate_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86acf733d80244159f20abf759fb3603" is_pre_election: true
I20260809 04:35:08.371410 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8ddf4243c9425c94e65f1b51ee6542" candidate_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8875775338674f0d82511974d1dbb1ca" is_pre_election: true
I20260809 04:35:08.371675 22501 raft_consensus.cc:2454] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5f1a218f4a4840b7fb5616d3b2ac35 in term 0.
I20260809 04:35:08.371675 22919 raft_consensus.cc:2454] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5f1a218f4a4840b7fb5616d3b2ac35 in term 0.
I20260809 04:35:08.372246 22611 leader_election.cc:302] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86acf733d80244159f20abf759fb3603, bf5f1a218f4a4840b7fb5616d3b2ac35; no voters:
I20260809 04:35:08.372560 23026 raft_consensus.cc:2781] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:08.372651 23026 raft_consensus.cc:482] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:08.372702 23026 raft_consensus.cc:3037] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.373819 23026 raft_consensus.cc:504] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.374191 23026 leader_election.cc:288] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [CANDIDATE]: Term 1 election: Requested vote from peers 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597), 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649)
I20260809 04:35:08.374653 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8ddf4243c9425c94e65f1b51ee6542" candidate_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86acf733d80244159f20abf759fb3603"
I20260809 04:35:08.374655 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8ddf4243c9425c94e65f1b51ee6542" candidate_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8875775338674f0d82511974d1dbb1ca"
I20260809 04:35:08.374805 22501 raft_consensus.cc:3037] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.374805 22919 raft_consensus.cc:3037] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.376240 22919 raft_consensus.cc:2454] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5f1a218f4a4840b7fb5616d3b2ac35 in term 1.
I20260809 04:35:08.376240 22501 raft_consensus.cc:2454] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5f1a218f4a4840b7fb5616d3b2ac35 in term 1.
I20260809 04:35:08.376642 22611 leader_election.cc:302] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86acf733d80244159f20abf759fb3603, bf5f1a218f4a4840b7fb5616d3b2ac35; no voters:
I20260809 04:35:08.376833 23026 raft_consensus.cc:2781] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:08.377028 23026 raft_consensus.cc:686] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 LEADER]: Becoming Leader. State: Replica: bf5f1a218f4a4840b7fb5616d3b2ac35, State: Running, Role: LEADER
I20260809 04:35:08.377444 23026 consensus_queue.cc:237] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.381206 22287 catalog_manager.cc:5495] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2). New cstate: current_term: 1 leader_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } health_report { overall_health: UNKNOWN } } }
Aug 09 04:35:08 dist-test-slave-rmhg krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250107, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:08.424063 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.415183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34885 (local address 127.0.0.1:58120)
0809 04:35:08.415519 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.415535 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.415656 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.416624 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.416626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.417050 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:08.417184 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.417191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.417946 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.417949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.418541 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.418551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.418944 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.418946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.419009 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.419595 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:08.419608 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:08.421007 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:08.422529 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:08.422539 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:08.422545 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:08.422772 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:08.423006 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:08.423008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:08.423009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:08.423081 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:08.423467 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:08.423475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:08.423587 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.423690 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.423856 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
I20260809 04:35:08.424063 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.415290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.19.92.62:34885)
0809 04:35:08.416415 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.416418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.416434 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:35:08.416470 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.416472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.416514 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.416584 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:08.417302 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.417834 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.418648 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.418860 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.418929 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.421138 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:08.421141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:08.421143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:08.421184 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:08.422420 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:08.422851 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:08.422854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:08.422855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:08.422943 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:08.423141 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:08.423143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:08.423144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:08.423344 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:08.423429 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.423750 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.423944 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":54}
I20260809 04:35:08.428192 22287 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:58120:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 04:35:08.428962 22287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:08.433766 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 99c3a7b898de4472afbbde9acc2f291b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55d9392e42c040b39614c52e48222055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:08.434053 22883 tablet_service.cc:1467] Processing CreateTablet for tablet 99c3a7b898de4472afbbde9acc2f291b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55d9392e42c040b39614c52e48222055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:08.434074 22675 tablet_service.cc:1467] Processing CreateTablet for tablet 99c3a7b898de4472afbbde9acc2f291b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55d9392e42c040b39614c52e48222055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:08.434549 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c3a7b898de4472afbbde9acc2f291b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:08.434604 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c3a7b898de4472afbbde9acc2f291b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:08.434755 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c3a7b898de4472afbbde9acc2f291b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:08.437381 23016 tablet_bootstrap.cc:492] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: Bootstrap starting.
I20260809 04:35:08.437467 23017 tablet_bootstrap.cc:492] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: Bootstrap starting.
I20260809 04:35:08.437505 23019 tablet_bootstrap.cc:492] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: Bootstrap starting.
I20260809 04:35:08.438156 23017 tablet_bootstrap.cc:654] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:08.438426 23019 tablet_bootstrap.cc:654] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:08.438709 23016 tablet_bootstrap.cc:654] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:08.439385 23017 tablet_bootstrap.cc:492] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: No bootstrap required, opened a new log
I20260809 04:35:08.439478 23017 ts_tablet_manager.cc:1397] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:08.439983 23017 raft_consensus.cc:348] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.440107 23017 raft_consensus.cc:374] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:08.440136 23019 tablet_bootstrap.cc:492] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: No bootstrap required, opened a new log
I20260809 04:35:08.440145 23017 raft_consensus.cc:729] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86acf733d80244159f20abf759fb3603, State: Initialized, Role: FOLLOWER
I20260809 04:35:08.440213 23019 ts_tablet_manager.cc:1397] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:08.440268 23017 consensus_queue.cc:260] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.440531 23016 tablet_bootstrap.cc:492] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: No bootstrap required, opened a new log
I20260809 04:35:08.440537 23017 ts_tablet_manager.cc:1428] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:08.440616 23016 ts_tablet_manager.cc:1397] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:08.440600 23025 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.440745 23025 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:08.440984 23016 raft_consensus.cc:348] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.440955 23019 raft_consensus.cc:348] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.441061 23016 raft_consensus.cc:374] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:08.441066 23019 raft_consensus.cc:374] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:08.441080 23016 raft_consensus.cc:729] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5f1a218f4a4840b7fb5616d3b2ac35, State: Initialized, Role: FOLLOWER
I20260809 04:35:08.441098 23019 raft_consensus.cc:729] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8875775338674f0d82511974d1dbb1ca, State: Initialized, Role: FOLLOWER
I20260809 04:35:08.441213 23016 consensus_queue.cc:260] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.441239 23019 consensus_queue.cc:260] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.441491 23016 ts_tablet_manager.cc:1428] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:08.441496 23019 ts_tablet_manager.cc:1428] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:08.441625 23027 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.441749 23027 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:08.441910 23026 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.442031 23026 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:08.688211 23027 raft_consensus.cc:482] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:08.688406 23027 raft_consensus.cc:504] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.689695 23027 leader_election.cc:288] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597), bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613)
I20260809 04:35:08.694801 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.690509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:48851 (local address 127.19.92.2:37613)
0809 04:35:08.690786 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.690801 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.690814 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:08.690858 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.690860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.690898 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.690973 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:08.691191 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.691910 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.693479 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.694103 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.694210 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.694282 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.694588 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.694663 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260809 04:35:08.694800 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.690078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:44351 (local address 127.19.92.1:38597)
0809 04:35:08.690224 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.690229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.690390 (+ 161us) server_negotiation.cc:407] Connection header received
0809 04:35:08.690543 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.690546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.690598 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.690671 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:08.690993 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.691785 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.693480 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.694101 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.694210 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.694282 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.694568 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.694653 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:35:08.694875 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.689978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38597 (local address 127.19.92.3:44351)
0809 04:35:08.690264 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.690278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.690423 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.690724 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.690727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.690755 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:08.690886 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.690892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.691907 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.691910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.693344 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.693356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.694256 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.694261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.694418 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.694455 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.694538 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.694663 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":83,"spinlock_wait_cycles":7808,"thread_start_us":97,"threads_started":1}
I20260809 04:35:08.694881 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.690365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:37613 (local address 127.19.92.3:48851)
0809 04:35:08.690661 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.690673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.690752 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.690997 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.690999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.691017 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:08.691100 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.691104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.692007 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.692010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.693344 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.693356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.694256 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.694261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.694421 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.694455 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.694557 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.694666 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":85,"spinlock_wait_cycles":1280,"thread_start_us":73,"threads_started":1}
I20260809 04:35:08.695492 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c3a7b898de4472afbbde9acc2f291b" candidate_uuid: "8875775338674f0d82511974d1dbb1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86acf733d80244159f20abf759fb3603" is_pre_election: true
I20260809 04:35:08.695657 22501 raft_consensus.cc:2454] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8875775338674f0d82511974d1dbb1ca in term 0.
I20260809 04:35:08.695607 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c3a7b898de4472afbbde9acc2f291b" candidate_uuid: "8875775338674f0d82511974d1dbb1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" is_pre_election: true
I20260809 04:35:08.695847 22707 raft_consensus.cc:2454] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8875775338674f0d82511974d1dbb1ca in term 0.
I20260809 04:35:08.696055 22809 leader_election.cc:302] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86acf733d80244159f20abf759fb3603, 8875775338674f0d82511974d1dbb1ca; no voters:
I20260809 04:35:08.696337 23027 raft_consensus.cc:2781] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:08.696399 23027 raft_consensus.cc:482] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:08.696426 23027 raft_consensus.cc:3037] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.697484 23027 raft_consensus.cc:504] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.697820 23027 leader_election.cc:288] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [CANDIDATE]: Term 1 election: Requested vote from peers 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597), bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613)
I20260809 04:35:08.698241 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c3a7b898de4472afbbde9acc2f291b" candidate_uuid: "8875775338674f0d82511974d1dbb1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35"
I20260809 04:35:08.698344 22707 raft_consensus.cc:3037] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.698611 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c3a7b898de4472afbbde9acc2f291b" candidate_uuid: "8875775338674f0d82511974d1dbb1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86acf733d80244159f20abf759fb3603"
I20260809 04:35:08.698729 22501 raft_consensus.cc:3037] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.699301 22707 raft_consensus.cc:2454] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8875775338674f0d82511974d1dbb1ca in term 1.
I20260809 04:35:08.699586 22809 leader_election.cc:302] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8875775338674f0d82511974d1dbb1ca, bf5f1a218f4a4840b7fb5616d3b2ac35; no voters:
I20260809 04:35:08.699740 22501 raft_consensus.cc:2454] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8875775338674f0d82511974d1dbb1ca in term 1.
I20260809 04:35:08.699750 23027 raft_consensus.cc:2781] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:08.699923 23027 raft_consensus.cc:686] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [term 1 LEADER]: Becoming Leader. State: Replica: 8875775338674f0d82511974d1dbb1ca, State: Running, Role: LEADER
I20260809 04:35:08.700251 23027 consensus_queue.cc:237] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.702425 23055 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: TxnStatusTablet state changed. Reason: New leader 8875775338674f0d82511974d1dbb1ca. Latest consensus state: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.702548 23055 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:08.702852 23060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 04:35:08.703099 23060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 04:35:08.703604 22289 catalog_manager.cc:5495] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8875775338674f0d82511974d1dbb1ca (127.19.92.3). New cstate: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } health_report { overall_health: HEALTHY } } }
I20260809 04:35:08.734717 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.730952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:34649 (local address 127.0.0.1:37678)
0809 04:35:08.731110 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.731126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.731237 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.731545 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.731548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.731556 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:08.731689 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.731696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.732442 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.732446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.733107 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.733117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.733578 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.733582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.733669 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.733714 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.734401 (+ 687us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.734418 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.734483 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.734587 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 04:35:08.734719 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.731096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.19.92.3:34649)
0809 04:35:08.731252 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.731257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.731271 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:08.731372 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.731375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.731431 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.731509 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:08.731798 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.732333 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.733249 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.733474 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.733566 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.733852 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.734305 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.734415 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.734505 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.734568 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:35:08.740046 22501 raft_consensus.cc:1261] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Refusing update from remote peer 8875775338674f0d82511974d1dbb1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:08.740047 22707 raft_consensus.cc:1261] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Refusing update from remote peer 8875775338674f0d82511974d1dbb1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:08.740724 23027 consensus_queue.cc:1035] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:08.740921 23055 consensus_queue.cc:1035] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:08.743945 23026 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: TxnStatusTablet state changed. Reason: New leader 8875775338674f0d82511974d1dbb1ca. Latest consensus state: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.744064 23026 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:08.744911 23025 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: TxnStatusTablet state changed. Reason: New leader 8875775338674f0d82511974d1dbb1ca. Latest consensus state: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.745982 23025 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:08.748232 23055 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.748350 23055 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:08.748472 23055 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.748556 23055 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P 8875775338674f0d82511974d1dbb1ca: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:08.748725 23026 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.748823 23026 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P bf5f1a218f4a4840b7fb5616d3b2ac35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:08.750281 23025 tablet_replica.cc:439] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } } }
I20260809 04:35:08.750383 23065 mvcc.cc:205] Tried to move back new op lower bound from 7316480445392068608 to 7316480445238861824. Current Snapshot: MvccSnapshot[applied={T|T < 7316480445392068608}]
I20260809 04:35:08.750386 23025 tablet_replica.cc:442] T 99c3a7b898de4472afbbde9acc2f291b P 86acf733d80244159f20abf759fb3603: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:08.755560 23064 mvcc.cc:205] Tried to move back new op lower bound from 7316480445392068608 to 7316480445238861824. Current Snapshot: MvccSnapshot[applied={T|T < 7316480445392068608}]
I20260809 04:35:08.759168 23026 consensus_queue.cc:1035] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:08.761826 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.757439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.19.92.2:37613)
0809 04:35:08.757552 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.757557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.757906 (+ 349us) server_negotiation.cc:407] Connection header received
0809 04:35:08.758116 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.758119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.758195 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.758310 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:08.758536 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.759047 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.760249 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.760560 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.760662 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.760930 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.761486 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.761593 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.761657 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.761711 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 04:35:08.768603 23031 consensus_queue.cc:1035] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:08.775664 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.771763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:34649 (local address 127.19.92.2:38903)
0809 04:35:08.772065 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.772080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.772177 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.772405 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.772408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.772416 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:08.772546 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.772553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.773594 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.773597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.774261 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.774271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.774833 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.774836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.774909 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.774980 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.775369 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.775377 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.775444 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.775525 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
I20260809 04:35:08.775661 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.772123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:38903 (local address 127.19.92.3:34649)
0809 04:35:08.772225 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.772229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.772240 (+ 11us) server_negotiation.cc:407] Connection header received
0809 04:35:08.772279 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.772282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.772323 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.772383 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:08.772649 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.773481 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.774373 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.774726 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.774831 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.775134 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.775231 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.775302 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.775468 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.775527 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 04:35:08.802554 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.798505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:37613 (local address 127.19.92.3:35951)
0809 04:35:08.798825 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.798840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.798983 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.799350 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.799353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.799361 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:08.799486 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.799492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.800183 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.800187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.801198 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.801211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.801791 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.801794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.801874 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.801929 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.802274 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.802280 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.802342 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.802428 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
I20260809 04:35:08.802557 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.798584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:35951 (local address 127.19.92.2:37613)
0809 04:35:08.798810 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.798814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.798906 (+ 92us) server_negotiation.cc:407] Connection header received
0809 04:35:08.799112 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.799115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.799167 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.799213 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:08.799578 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.800083 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.801312 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.801624 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.801720 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.802083 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.802169 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.802210 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.802379 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.802434 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260809 04:35:08.825727 22287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:08.825906 22287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:08.828317 22287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: Sending DeleteTablet for 3 replicas of tablet 6a8ddf4243c9425c94e65f1b51ee6542
I20260809 04:35:08.829188 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a8ddf4243c9425c94e65f1b51ee6542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:08 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:46266
I20260809 04:35:08.829511 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a8ddf4243c9425c94e65f1b51ee6542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:08 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:37666
I20260809 04:35:08.830013 23098 tablet_replica.cc:330] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: stopping tablet replica
I20260809 04:35:08.830053 22675 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a8ddf4243c9425c94e65f1b51ee6542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:08 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:53400
I20260809 04:35:08.830375 23098 raft_consensus.cc:2229] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:08.830608 23098 raft_consensus.cc:2258] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:08.831218 22287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:08.831655 22287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:08.832348 23098 ts_tablet_manager.cc:1905] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:08.833451 23095 tablet_replica.cc:330] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: stopping tablet replica
I20260809 04:35:08.833688 23100 tablet_replica.cc:330] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: stopping tablet replica
I20260809 04:35:08.834017 23100 raft_consensus.cc:2229] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:08.834348 23095 raft_consensus.cc:2229] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:08.834754 23100 raft_consensus.cc:2258] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:08.835008 23095 raft_consensus.cc:2258] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:08.837817 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f (DEFAULT_TABLE table=test-table [id=41ec410a61a5481c8e1bdea02141f3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:08.838109 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:08.840654 22675 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f (DEFAULT_TABLE table=test-table [id=41ec410a61a5481c8e1bdea02141f3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:08.840907 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:08.841085 23098 ts_tablet_manager.cc:1918] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:08.841221 23098 log.cc:1196] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/6a8ddf4243c9425c94e65f1b51ee6542
I20260809 04:35:08.841979 23098 ts_tablet_manager.cc:1939] T 6a8ddf4243c9425c94e65f1b51ee6542 P 8875775338674f0d82511974d1dbb1ca: Deleting consensus metadata
I20260809 04:35:08.843488 22883 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f (DEFAULT_TABLE table=test-table [id=41ec410a61a5481c8e1bdea02141f3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:08.843734 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:08.845331 23016 tablet_bootstrap.cc:492] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: Bootstrap starting.
I20260809 04:35:08.845564 23019 tablet_bootstrap.cc:492] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: Bootstrap starting.
I20260809 04:35:08.846979 23016 tablet_bootstrap.cc:654] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:08.847597 23019 tablet_bootstrap.cc:654] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:08.848357 23017 tablet_bootstrap.cc:492] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: Bootstrap starting.
I20260809 04:35:08.849601 23019 tablet_bootstrap.cc:492] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: No bootstrap required, opened a new log
I20260809 04:35:08.849690 23019 ts_tablet_manager.cc:1397] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260809 04:35:08.850351 23019 raft_consensus.cc:348] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.850561 23019 raft_consensus.cc:374] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:08.850606 23019 raft_consensus.cc:729] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8875775338674f0d82511974d1dbb1ca, State: Initialized, Role: FOLLOWER
I20260809 04:35:08.850744 23019 consensus_queue.cc:260] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.851037 23019 ts_tablet_manager.cc:1428] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:08.851444 23017 tablet_bootstrap.cc:654] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:08.851425 22266 catalog_manager.cc:4841] TS 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649): tablet 6a8ddf4243c9425c94e65f1b51ee6542 (table test-table [id=a8765fc7762b4220ad4600e94df96b39]) successfully deleted
I20260809 04:35:08.854943 23100 ts_tablet_manager.cc:1905] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:08.856854 23095 ts_tablet_manager.cc:1905] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:08.858263 23016 tablet_bootstrap.cc:492] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: No bootstrap required, opened a new log
I20260809 04:35:08.858379 23016 ts_tablet_manager.cc:1397] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260809 04:35:08.859321 23016 raft_consensus.cc:348] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.859447 23016 raft_consensus.cc:374] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:08.859483 23016 raft_consensus.cc:729] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5f1a218f4a4840b7fb5616d3b2ac35, State: Initialized, Role: FOLLOWER
I20260809 04:35:08.859627 23016 consensus_queue.cc:260] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.859920 23016 ts_tablet_manager.cc:1428] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 04:35:08.863653 23100 ts_tablet_manager.cc:1918] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:08.863759 23100 log.cc:1196] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/6a8ddf4243c9425c94e65f1b51ee6542
I20260809 04:35:08.864616 23017 tablet_bootstrap.cc:492] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: No bootstrap required, opened a new log
I20260809 04:35:08.864724 23017 ts_tablet_manager.cc:1397] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20260809 04:35:08.865533 23017 raft_consensus.cc:348] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.865648 23017 raft_consensus.cc:374] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:08.865675 23017 raft_consensus.cc:729] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86acf733d80244159f20abf759fb3603, State: Initialized, Role: FOLLOWER
I20260809 04:35:08.865790 23017 consensus_queue.cc:260] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.866055 23017 ts_tablet_manager.cc:1428] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:08.866393 23095 ts_tablet_manager.cc:1918] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:08.866487 23095 log.cc:1196] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/6a8ddf4243c9425c94e65f1b51ee6542
I20260809 04:35:08.866839 23095 ts_tablet_manager.cc:1939] T 6a8ddf4243c9425c94e65f1b51ee6542 P 86acf733d80244159f20abf759fb3603: Deleting consensus metadata
I20260809 04:35:08.867198 23100 ts_tablet_manager.cc:1939] T 6a8ddf4243c9425c94e65f1b51ee6542 P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting consensus metadata
I20260809 04:35:08.868520 22270 catalog_manager.cc:4841] TS bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613): tablet 6a8ddf4243c9425c94e65f1b51ee6542 (table test-table [id=a8765fc7762b4220ad4600e94df96b39]) successfully deleted
I20260809 04:35:08.869427 22270 catalog_manager.cc:4841] TS 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597): tablet 6a8ddf4243c9425c94e65f1b51ee6542 (table test-table [id=a8765fc7762b4220ad4600e94df96b39]) successfully deleted
I20260809 04:35:08.876096 23055 raft_consensus.cc:482] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:08.876228 23055 raft_consensus.cc:504] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.876786 23055 leader_election.cc:288] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597), bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613)
I20260809 04:35:08.877249 22705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd3a8b4ec05470ba4c9f747e0da8c8f" candidate_uuid: "8875775338674f0d82511974d1dbb1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" is_pre_election: true
I20260809 04:35:08.877249 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd3a8b4ec05470ba4c9f747e0da8c8f" candidate_uuid: "8875775338674f0d82511974d1dbb1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86acf733d80244159f20abf759fb3603" is_pre_election: true
I20260809 04:35:08.877395 22501 raft_consensus.cc:2454] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8875775338674f0d82511974d1dbb1ca in term 0.
I20260809 04:35:08.877396 22705 raft_consensus.cc:2454] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8875775338674f0d82511974d1dbb1ca in term 0.
I20260809 04:35:08.877727 22809 leader_election.cc:302] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8875775338674f0d82511974d1dbb1ca, bf5f1a218f4a4840b7fb5616d3b2ac35; no voters:
I20260809 04:35:08.877902 23055 raft_consensus.cc:2781] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:08.877972 23055 raft_consensus.cc:482] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:08.878011 23055 raft_consensus.cc:3037] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.879174 23055 raft_consensus.cc:504] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.879549 23055 leader_election.cc:288] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [CANDIDATE]: Term 1 election: Requested vote from peers 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597), bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613)
I20260809 04:35:08.880061 22705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd3a8b4ec05470ba4c9f747e0da8c8f" candidate_uuid: "8875775338674f0d82511974d1dbb1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35"
I20260809 04:35:08.880180 22705 raft_consensus.cc:3037] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.880457 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd3a8b4ec05470ba4c9f747e0da8c8f" candidate_uuid: "8875775338674f0d82511974d1dbb1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86acf733d80244159f20abf759fb3603"
I20260809 04:35:08.880591 22501 raft_consensus.cc:3037] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:08.881743 22501 raft_consensus.cc:2454] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8875775338674f0d82511974d1dbb1ca in term 1.
I20260809 04:35:08.882119 22809 leader_election.cc:302] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86acf733d80244159f20abf759fb3603, 8875775338674f0d82511974d1dbb1ca; no voters:
I20260809 04:35:08.882226 22705 raft_consensus.cc:2454] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8875775338674f0d82511974d1dbb1ca in term 1.
I20260809 04:35:08.882844 23055 raft_consensus.cc:2781] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:08.882931 23055 raft_consensus.cc:686] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 1 LEADER]: Becoming Leader. State: Replica: 8875775338674f0d82511974d1dbb1ca, State: Running, Role: LEADER
I20260809 04:35:08.883071 23055 consensus_queue.cc:237] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:08.884979 22287 catalog_manager.cc:5495] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8875775338674f0d82511974d1dbb1ca (127.19.92.3). New cstate: current_term: 1 leader_uuid: "8875775338674f0d82511974d1dbb1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } health_report { overall_health: HEALTHY } } }
I20260809 04:35:08.906988 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.903156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.19.92.3:34649)
0809 04:35:08.903254 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.903258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.903271 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:08.903429 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.903432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.903483 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.903555 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:08.903805 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.904696 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.905905 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.906204 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.906285 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.906589 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.906664 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.906726 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.906838 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.906882 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 04:35:08.915957 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.912612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:34649 (local address 127.19.92.3:45665)
0809 04:35:08.912725 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:08.912739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:08.912804 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:08.913056 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:08.913059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:08.913065 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:08.913204 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.913211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.914194 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.914197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.914896 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:08.914907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.915420 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.915422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:08.915489 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.915513 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.915742 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.915746 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:35:08.915782 (+ 36us) client_negotiation.cc:240] Negotiation successful
0809 04:35:08.915844 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":18}
I20260809 04:35:08.916007 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:08.912674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:45665 (local address 127.19.92.3:34649)
0809 04:35:08.912780 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:08.912784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:08.912798 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:08.912913 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:08.912916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:08.912973 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:08.913039 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:08.913295 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.914096 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.914994 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:08.915329 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:08.915431 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:08.915604 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:08.915679 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:08.915744 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:08.915807 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 04:35:08.915855 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":42}
I20260809 04:35:08.921850 22501 raft_consensus.cc:1261] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Refusing update from remote peer 8875775338674f0d82511974d1dbb1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:08.921983 22707 raft_consensus.cc:1261] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Refusing update from remote peer 8875775338674f0d82511974d1dbb1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:08.922326 23055 consensus_queue.cc:1035] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:08.922451 23071 consensus_queue.cc:1035] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:08.927300 23063 mvcc.cc:205] Tried to move back new op lower bound from 7316480446140723200 to 7316480445986562048. Current Snapshot: MvccSnapshot[applied={T|T < 7316480446140723200}]
W20260809 04:35:08.938387 23063 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 04:35:10.494225 22287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:10.494390 22287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:10.497000 22287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: Sending DeleteTablet for 3 replicas of tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f
I20260809 04:35:10.497727 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:10 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:37666
I20260809 04:35:10.497737 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:10 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:46266
I20260809 04:35:10.497737 22675 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:10 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:53400
I20260809 04:35:10.498145 23136 tablet_replica.cc:330] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: stopping tablet replica
I20260809 04:35:10.498230 23137 tablet_replica.cc:330] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: stopping tablet replica
I20260809 04:35:10.498256 23136 raft_consensus.cc:2229] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:10.498545 23136 raft_consensus.cc:2258] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:10.498577 23137 raft_consensus.cc:2229] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:10.498708 23137 raft_consensus.cc:2258] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:10.499217 23136 ts_tablet_manager.cc:1905] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:10.500818 23135 tablet_replica.cc:330] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: stopping tablet replica
I20260809 04:35:10.501358 23137 ts_tablet_manager.cc:1905] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:10.501682 23136 ts_tablet_manager.cc:1918] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:35:10.501796 23136 log.cc:1196] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/0dd3a8b4ec05470ba4c9f747e0da8c8f
I20260809 04:35:10.502107 23136 ts_tablet_manager.cc:1939] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting consensus metadata
I20260809 04:35:10.502779 23135 raft_consensus.cc:2229] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:10.502856 22270 catalog_manager.cc:4841] TS bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613): tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f (table test-table [id=41ec410a61a5481c8e1bdea02141f3fc]) successfully deleted
I20260809 04:35:10.503187 23135 raft_consensus.cc:2258] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:10.503846 23135 ts_tablet_manager.cc:1905] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:10.504046 23137 ts_tablet_manager.cc:1918] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:35:10.504114 23137 log.cc:1196] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/0dd3a8b4ec05470ba4c9f747e0da8c8f
I20260809 04:35:10.504352 23137 ts_tablet_manager.cc:1939] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 86acf733d80244159f20abf759fb3603: Deleting consensus metadata
I20260809 04:35:10.505100 22270 catalog_manager.cc:4841] TS 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597): tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f (table test-table [id=41ec410a61a5481c8e1bdea02141f3fc]) successfully deleted
I20260809 04:35:10.506399 23135 ts_tablet_manager.cc:1918] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:35:10.506469 23135 log.cc:1196] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/0dd3a8b4ec05470ba4c9f747e0da8c8f
I20260809 04:35:10.506723 23135 ts_tablet_manager.cc:1939] T 0dd3a8b4ec05470ba4c9f747e0da8c8f P 8875775338674f0d82511974d1dbb1ca: Deleting consensus metadata
I20260809 04:35:10.507359 22266 catalog_manager.cc:4841] TS 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649): tablet 0dd3a8b4ec05470ba4c9f747e0da8c8f (table test-table [id=41ec410a61a5481c8e1bdea02141f3fc]) successfully deleted
Aug 09 04:35:10 dist-test-slave-rmhg krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:35:10 dist-test-slave-rmhg krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:10.526719 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.517781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.19.92.62:34885)
0809 04:35:10.518090 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:10.518099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:10.518117 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:35:10.518283 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:10.518286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:10.518343 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:10.518422 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:10.519382 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.519998 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.520863 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.521117 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.521252 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.523690 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:10.523694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:10.523696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:10.523724 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:10.525012 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:10.525490 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:10.525493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:10.525495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:10.525582 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:10.525842 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:10.525846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:10.525848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:10.526047 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:10.526168 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:10.526409 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 04:35:10.526549 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260809 04:35:10.529373 22287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:10.529843 22287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:10.534646 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc008d5d64c4694b098e99a7772bb03 (DEFAULT_TABLE table=test-table [id=6bd10391ea45486fbf614801f316fdb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:10.534660 22675 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc008d5d64c4694b098e99a7772bb03 (DEFAULT_TABLE table=test-table [id=6bd10391ea45486fbf614801f316fdb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:10.534729 22883 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc008d5d64c4694b098e99a7772bb03 (DEFAULT_TABLE table=test-table [id=6bd10391ea45486fbf614801f316fdb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:10.535004 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc008d5d64c4694b098e99a7772bb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:10.535004 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc008d5d64c4694b098e99a7772bb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:10.535015 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc008d5d64c4694b098e99a7772bb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:10.537448 23146 tablet_bootstrap.cc:492] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: Bootstrap starting.
I20260809 04:35:10.538130 23149 tablet_bootstrap.cc:492] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: Bootstrap starting.
I20260809 04:35:10.538172 23148 tablet_bootstrap.cc:492] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: Bootstrap starting.
I20260809 04:35:10.538388 23146 tablet_bootstrap.cc:654] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:10.539134 23149 tablet_bootstrap.cc:654] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:10.539498 23146 tablet_bootstrap.cc:492] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: No bootstrap required, opened a new log
I20260809 04:35:10.539577 23148 tablet_bootstrap.cc:654] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:10.539587 23146 ts_tablet_manager.cc:1397] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:10.540031 23146 raft_consensus.cc:348] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.540167 23146 raft_consensus.cc:374] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:10.540205 23146 raft_consensus.cc:729] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5f1a218f4a4840b7fb5616d3b2ac35, State: Initialized, Role: FOLLOWER
I20260809 04:35:10.540362 23146 consensus_queue.cc:260] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.540793 23146 ts_tablet_manager.cc:1428] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:10.541535 23148 tablet_bootstrap.cc:492] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: No bootstrap required, opened a new log
I20260809 04:35:10.541575 23149 tablet_bootstrap.cc:492] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: No bootstrap required, opened a new log
I20260809 04:35:10.541638 23148 ts_tablet_manager.cc:1397] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 04:35:10.541647 23149 ts_tablet_manager.cc:1397] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 04:35:10.542055 23149 raft_consensus.cc:348] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.542061 23148 raft_consensus.cc:348] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.542169 23148 raft_consensus.cc:374] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:10.542169 23149 raft_consensus.cc:374] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:10.542202 23149 raft_consensus.cc:729] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8875775338674f0d82511974d1dbb1ca, State: Initialized, Role: FOLLOWER
I20260809 04:35:10.542201 23148 raft_consensus.cc:729] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86acf733d80244159f20abf759fb3603, State: Initialized, Role: FOLLOWER
I20260809 04:35:10.542332 23149 consensus_queue.cc:260] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.542349 23148 consensus_queue.cc:260] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.542610 23149 ts_tablet_manager.cc:1428] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:10.542773 23148 ts_tablet_manager.cc:1428] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 04:35:10.630968 23155 raft_consensus.cc:482] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:10.631136 23155 raft_consensus.cc:504] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.631747 23155 leader_election.cc:288] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597), 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649)
I20260809 04:35:10.632086 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc008d5d64c4694b098e99a7772bb03" candidate_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86acf733d80244159f20abf759fb3603" is_pre_election: true
I20260809 04:35:10.632087 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc008d5d64c4694b098e99a7772bb03" candidate_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8875775338674f0d82511974d1dbb1ca" is_pre_election: true
I20260809 04:35:10.632252 22919 raft_consensus.cc:2454] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5f1a218f4a4840b7fb5616d3b2ac35 in term 0.
I20260809 04:35:10.632252 22501 raft_consensus.cc:2454] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5f1a218f4a4840b7fb5616d3b2ac35 in term 0.
I20260809 04:35:10.632591 22606 leader_election.cc:302] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8875775338674f0d82511974d1dbb1ca, bf5f1a218f4a4840b7fb5616d3b2ac35; no voters:
I20260809 04:35:10.632817 23155 raft_consensus.cc:2781] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:10.632880 23155 raft_consensus.cc:482] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:10.632942 23155 raft_consensus.cc:3037] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:10.634006 23155 raft_consensus.cc:504] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.634325 23155 leader_election.cc:288] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [CANDIDATE]: Term 1 election: Requested vote from peers 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597), 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649)
I20260809 04:35:10.634696 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc008d5d64c4694b098e99a7772bb03" candidate_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86acf733d80244159f20abf759fb3603"
I20260809 04:35:10.634692 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc008d5d64c4694b098e99a7772bb03" candidate_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8875775338674f0d82511974d1dbb1ca"
I20260809 04:35:10.634816 22919 raft_consensus.cc:3037] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:10.634816 22501 raft_consensus.cc:3037] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:10.635938 22501 raft_consensus.cc:2454] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5f1a218f4a4840b7fb5616d3b2ac35 in term 1.
I20260809 04:35:10.635946 22919 raft_consensus.cc:2454] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5f1a218f4a4840b7fb5616d3b2ac35 in term 1.
I20260809 04:35:10.636219 22606 leader_election.cc:302] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8875775338674f0d82511974d1dbb1ca, bf5f1a218f4a4840b7fb5616d3b2ac35; no voters:
I20260809 04:35:10.636436 23155 raft_consensus.cc:2781] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:10.636667 23155 raft_consensus.cc:686] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 LEADER]: Becoming Leader. State: Replica: bf5f1a218f4a4840b7fb5616d3b2ac35, State: Running, Role: LEADER
I20260809 04:35:10.636807 23155 consensus_queue.cc:237] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.638432 22289 catalog_manager.cc:5495] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2). New cstate: current_term: 1 leader_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } health_report { overall_health: UNKNOWN } } }
I20260809 04:35:10.670136 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.667022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.19.92.2:37613)
0809 04:35:10.667293 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:10.667300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:10.667312 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:35:10.667350 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:10.667352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:10.667393 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:10.667565 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:10.667797 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.668304 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.669129 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.669369 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.669437 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.669666 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:10.669736 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:10.669786 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:10.669959 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 04:35:10.670003 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":128,"threads_started":1}
I20260809 04:35:10.676846 22919 raft_consensus.cc:1261] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Refusing update from remote peer bf5f1a218f4a4840b7fb5616d3b2ac35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:10.677248 22501 raft_consensus.cc:1261] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Refusing update from remote peer bf5f1a218f4a4840b7fb5616d3b2ac35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:10.677380 23161 consensus_queue.cc:1035] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:10.677620 23155 consensus_queue.cc:1035] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:10.709625 22287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:10.709782 22287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:10.711687 22287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: Sending DeleteTablet for 3 replicas of tablet 2cc008d5d64c4694b098e99a7772bb03
I20260809 04:35:10.712249 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc008d5d64c4694b098e99a7772bb03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:10 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:46266
I20260809 04:35:10.712250 22675 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc008d5d64c4694b098e99a7772bb03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:10 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:53400
I20260809 04:35:10.712304 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc008d5d64c4694b098e99a7772bb03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:10 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:37666
I20260809 04:35:10.712432 23136 tablet_replica.cc:330] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: stopping tablet replica
I20260809 04:35:10.712508 23137 tablet_replica.cc:330] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: stopping tablet replica
I20260809 04:35:10.712581 23136 raft_consensus.cc:2229] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:10.712607 23137 raft_consensus.cc:2229] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:10.712698 23137 raft_consensus.cc:2258] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:10.712771 23136 raft_consensus.cc:2258] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:10.712960 23135 tablet_replica.cc:330] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: stopping tablet replica
I20260809 04:35:10.713071 23135 raft_consensus.cc:2229] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:10.713210 23135 raft_consensus.cc:2258] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:10.713177 22287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 04:35:10.713503 23137 ts_tablet_manager.cc:1905] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 04:35:10.713569 22287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:10.714282 23136 ts_tablet_manager.cc:1905] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:10.714282 23135 ts_tablet_manager.cc:1905] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:10.716444 23137 ts_tablet_manager.cc:1918] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:10.716519 23137 log.cc:1196] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/2cc008d5d64c4694b098e99a7772bb03
I20260809 04:35:10.716773 23137 ts_tablet_manager.cc:1939] T 2cc008d5d64c4694b098e99a7772bb03 P 86acf733d80244159f20abf759fb3603: Deleting consensus metadata
I20260809 04:35:10.716836 23136 ts_tablet_manager.cc:1918] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:10.716905 23136 log.cc:1196] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/2cc008d5d64c4694b098e99a7772bb03
I20260809 04:35:10.716989 23135 ts_tablet_manager.cc:1918] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:10.717058 23135 log.cc:1196] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/2cc008d5d64c4694b098e99a7772bb03
I20260809 04:35:10.717170 23136 ts_tablet_manager.cc:1939] T 2cc008d5d64c4694b098e99a7772bb03 P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting consensus metadata
I20260809 04:35:10.717499 23135 ts_tablet_manager.cc:1939] T 2cc008d5d64c4694b098e99a7772bb03 P 8875775338674f0d82511974d1dbb1ca: Deleting consensus metadata
I20260809 04:35:10.717545 22270 catalog_manager.cc:4841] TS 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597): tablet 2cc008d5d64c4694b098e99a7772bb03 (table test-table [id=6bd10391ea45486fbf614801f316fdb0]) successfully deleted
I20260809 04:35:10.718343 22266 catalog_manager.cc:4841] TS 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649): tablet 2cc008d5d64c4694b098e99a7772bb03 (table test-table [id=6bd10391ea45486fbf614801f316fdb0]) successfully deleted
I20260809 04:35:10.718420 22472 tablet_service.cc:1467] Processing CreateTablet for tablet e16f2949f44947b7b49996143982c3b3 (DEFAULT_TABLE table=test-table [id=e8d1a8412ab840a3808159b457d6a302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:10.718606 22675 tablet_service.cc:1467] Processing CreateTablet for tablet e16f2949f44947b7b49996143982c3b3 (DEFAULT_TABLE table=test-table [id=e8d1a8412ab840a3808159b457d6a302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:10.718672 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16f2949f44947b7b49996143982c3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:10.718681 22883 tablet_service.cc:1467] Processing CreateTablet for tablet e16f2949f44947b7b49996143982c3b3 (DEFAULT_TABLE table=test-table [id=e8d1a8412ab840a3808159b457d6a302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:10.718812 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16f2949f44947b7b49996143982c3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:10.718923 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16f2949f44947b7b49996143982c3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:10.720777 23149 tablet_bootstrap.cc:492] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca: Bootstrap starting.
I20260809 04:35:10.720885 23148 tablet_bootstrap.cc:492] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603: Bootstrap starting.
I20260809 04:35:10.721571 23149 tablet_bootstrap.cc:654] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:10.722112 23148 tablet_bootstrap.cc:654] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:10.722584 23149 tablet_bootstrap.cc:492] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca: No bootstrap required, opened a new log
I20260809 04:35:10.722677 23149 ts_tablet_manager.cc:1397] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 04:35:10.723042 23149 raft_consensus.cc:348] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.723136 23149 raft_consensus.cc:374] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:10.723166 23149 raft_consensus.cc:729] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8875775338674f0d82511974d1dbb1ca, State: Initialized, Role: FOLLOWER
I20260809 04:35:10.723294 23149 consensus_queue.cc:260] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.723367 23146 tablet_bootstrap.cc:492] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35: Bootstrap starting.
I20260809 04:35:10.723534 23149 ts_tablet_manager.cc:1428] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 04:35:10.724155 23146 tablet_bootstrap.cc:654] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:10.724892 22270 catalog_manager.cc:4841] TS bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613): tablet 2cc008d5d64c4694b098e99a7772bb03 (table test-table [id=6bd10391ea45486fbf614801f316fdb0]) successfully deleted
I20260809 04:35:10.725412 23146 tablet_bootstrap.cc:492] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35: No bootstrap required, opened a new log
I20260809 04:35:10.725508 23146 ts_tablet_manager.cc:1397] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:10.725862 23146 raft_consensus.cc:348] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.725993 23146 raft_consensus.cc:374] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:10.726037 23146 raft_consensus.cc:729] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5f1a218f4a4840b7fb5616d3b2ac35, State: Initialized, Role: FOLLOWER
I20260809 04:35:10.726173 23146 consensus_queue.cc:260] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.726455 23146 ts_tablet_manager.cc:1428] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 04:35:10.727128 23148 tablet_bootstrap.cc:492] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603: No bootstrap required, opened a new log
I20260809 04:35:10.727231 23148 ts_tablet_manager.cc:1397] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260809 04:35:10.727957 23148 raft_consensus.cc:348] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.728077 23148 raft_consensus.cc:374] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:10.728122 23148 raft_consensus.cc:729] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86acf733d80244159f20abf759fb3603, State: Initialized, Role: FOLLOWER
I20260809 04:35:10.728253 23148 consensus_queue.cc:260] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.728528 23148 ts_tablet_manager.cc:1428] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 04:35:10.734587 23159 raft_consensus.cc:482] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:10.734771 23159 raft_consensus.cc:504] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.735901 23159 leader_election.cc:288] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613), 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649)
I20260809 04:35:10.741103 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.736291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:50437 (local address 127.19.92.2:37613)
0809 04:35:10.736418 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:10.736433 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:10.736535 (+ 102us) server_negotiation.cc:407] Connection header received
0809 04:35:10.736683 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:10.736686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:10.736736 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:10.736823 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:10.737403 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.738130 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.739988 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.740411 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.740509 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.740575 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:10.740910 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 04:35:10.740999 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 04:35:10.741232 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.736109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:37613 (local address 127.19.92.1:50437)
0809 04:35:10.736437 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:10.736453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:10.736578 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:10.736844 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:10.736847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:10.736882 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:10.737286 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:10.737295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.738354 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.738357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:10.739892 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:10.739900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.740635 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.740639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:10.740736 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.740810 (+ 74us) client_negotiation.cc:769] Sending connection context
0809 04:35:10.740885 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 04:35:10.741004 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":129,"spinlock_wait_cycles":16128,"thread_start_us":86,"threads_started":1}
I20260809 04:35:10.741271 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.736109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:34649 (local address 127.19.92.1:46103)
0809 04:35:10.736540 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:10.736552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:10.736622 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:10.736844 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:10.736847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:10.736876 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:10.737111 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:10.737119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.738194 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.738197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:10.739787 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:10.739799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.740501 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.740505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:10.740764 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.740808 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 04:35:10.740885 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:35:10.741005 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":182,"spinlock_wait_cycles":7808,"thread_start_us":114,"threads_started":1}
I20260809 04:35:10.741546 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.736207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:46103 (local address 127.19.92.3:34649)
0809 04:35:10.736562 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:10.736636 (+ 74us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:10.736652 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:35:10.736697 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:10.736700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:10.736742 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:10.736824 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:10.737242 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.738069 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.739916 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.740378 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.740470 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.740526 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:10.741316 (+ 790us) server_negotiation.cc:299] Negotiation successful
0809 04:35:10.741411 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":71,"threads_started":1}
I20260809 04:35:10.741676 22702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16f2949f44947b7b49996143982c3b3" candidate_uuid: "86acf733d80244159f20abf759fb3603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" is_pre_election: true
I20260809 04:35:10.741832 22702 raft_consensus.cc:2454] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86acf733d80244159f20abf759fb3603 in term 0.
I20260809 04:35:10.741935 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16f2949f44947b7b49996143982c3b3" candidate_uuid: "86acf733d80244159f20abf759fb3603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8875775338674f0d82511974d1dbb1ca" is_pre_election: true
I20260809 04:35:10.742082 22919 raft_consensus.cc:2454] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86acf733d80244159f20abf759fb3603 in term 0.
I20260809 04:35:10.742225 22408 leader_election.cc:302] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86acf733d80244159f20abf759fb3603, bf5f1a218f4a4840b7fb5616d3b2ac35; no voters:
I20260809 04:35:10.742520 23159 raft_consensus.cc:2781] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:10.742591 23159 raft_consensus.cc:482] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:10.742628 23159 raft_consensus.cc:3037] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:10.743656 23159 raft_consensus.cc:504] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.743969 23159 leader_election.cc:288] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [CANDIDATE]: Term 1 election: Requested vote from peers bf5f1a218f4a4840b7fb5616d3b2ac35 (127.19.92.2:37613), 8875775338674f0d82511974d1dbb1ca (127.19.92.3:34649)
I20260809 04:35:10.744311 22702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16f2949f44947b7b49996143982c3b3" candidate_uuid: "86acf733d80244159f20abf759fb3603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35"
I20260809 04:35:10.744311 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16f2949f44947b7b49996143982c3b3" candidate_uuid: "86acf733d80244159f20abf759fb3603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8875775338674f0d82511974d1dbb1ca"
I20260809 04:35:10.744442 22702 raft_consensus.cc:3037] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:10.744443 22919 raft_consensus.cc:3037] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:10.745687 22702 raft_consensus.cc:2454] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86acf733d80244159f20abf759fb3603 in term 1.
I20260809 04:35:10.745823 22919 raft_consensus.cc:2454] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86acf733d80244159f20abf759fb3603 in term 1.
I20260809 04:35:10.745971 22408 leader_election.cc:302] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86acf733d80244159f20abf759fb3603, bf5f1a218f4a4840b7fb5616d3b2ac35; no voters:
I20260809 04:35:10.746166 23159 raft_consensus.cc:2781] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:10.746385 23159 raft_consensus.cc:686] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [term 1 LEADER]: Becoming Leader. State: Replica: 86acf733d80244159f20abf759fb3603, State: Running, Role: LEADER
I20260809 04:35:10.746625 23159 consensus_queue.cc:237] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } }
I20260809 04:35:10.748765 22287 catalog_manager.cc:5495] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86acf733d80244159f20abf759fb3603 (127.19.92.1). New cstate: current_term: 1 leader_uuid: "86acf733d80244159f20abf759fb3603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86acf733d80244159f20abf759fb3603" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 } health_report { overall_health: UNKNOWN } } }
I20260809 04:35:10.764807 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.761170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.19.92.1:38597)
0809 04:35:10.761453 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:10.761461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:10.761474 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:10.761537 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:10.761540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:10.761588 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:10.761694 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:10.761914 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.762472 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.763310 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.763542 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.763629 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.763893 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:10.764404 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:10.764522 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:10.764585 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 04:35:10.764641 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
I20260809 04:35:10.774972 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.771741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:46451 (local address 127.19.92.3:34649)
0809 04:35:10.771863 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:10.771867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:10.771904 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:35:10.772021 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:10.772024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:10.772067 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:10.772130 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:10.772752 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.773292 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.774087 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.774293 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.774356 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.774614 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:10.774691 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:10.774738 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:10.774801 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 04:35:10.774851 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:35:10.774986 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.771512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:34649 (local address 127.19.92.1:46451)
0809 04:35:10.771847 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:10.771859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:10.771945 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:10.772138 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:10.772140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:10.772146 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:10.772660 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:10.772666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.773384 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.773386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:10.774000 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:10.774009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.774384 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.774386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:10.774453 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.774508 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:10.774747 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:10.774751 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:35:10.774788 (+ 37us) client_negotiation.cc:240] Negotiation successful
0809 04:35:10.774855 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":117,"threads_started":1}
I20260809 04:35:10.781486 22919 raft_consensus.cc:1261] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Refusing update from remote peer 86acf733d80244159f20abf759fb3603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:10.781514 22702 raft_consensus.cc:1261] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Refusing update from remote peer 86acf733d80244159f20abf759fb3603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:10.781937 23159 consensus_queue.cc:1035] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5f1a218f4a4840b7fb5616d3b2ac35" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:10.782130 23182 consensus_queue.cc:1035] T e16f2949f44947b7b49996143982c3b3 P 86acf733d80244159f20abf759fb3603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8875775338674f0d82511974d1dbb1ca" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:10.799386 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.795997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:37455 (local address 127.19.92.1:38597)
0809 04:35:10.796106 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:10.796111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:10.796420 (+ 309us) server_negotiation.cc:407] Connection header received
0809 04:35:10.796514 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:10.796517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:10.796556 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:10.796615 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:10.796876 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.797455 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.798273 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.798521 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.798623 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.798961 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:10.799031 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:10.799083 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:10.799210 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 04:35:10.799260 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 04:35:10.799424 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:10.795942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38597 (local address 127.19.92.3:37455)
0809 04:35:10.796327 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:10.796340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:10.796432 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:10.796640 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:10.796643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:10.796651 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:10.796769 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:10.796776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.797558 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.797561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:10.798167 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:10.798175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:10.798619 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:10.798621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:10.798704 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:10.798740 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:10.799110 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:10.799117 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:35:10.799184 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 04:35:10.799267 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":201,"threads_started":1}
W20260809 04:35:10.800305 23115 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 04:35:12.361340 22287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:12.361528 22287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:12.363804 22287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f07fbaf1e6784cada7a281c129a5b314: Sending DeleteTablet for 3 replicas of tablet e16f2949f44947b7b49996143982c3b3
I20260809 04:35:12.364377 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet e16f2949f44947b7b49996143982c3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:46266
I20260809 04:35:12.364562 22675 tablet_service.cc:1514] Processing DeleteTablet for tablet e16f2949f44947b7b49996143982c3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:53400
I20260809 04:35:12.364856 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 22385
I20260809 04:35:12.364861 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet e16f2949f44947b7b49996143982c3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:37666
I20260809 04:35:12.365121 23228 tablet_replica.cc:330] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35: stopping tablet replica
I20260809 04:35:12.372988 23228 raft_consensus.cc:2229] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:12.373211 23228 raft_consensus.cc:2258] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:12.373963 23228 ts_tablet_manager.cc:1905] T e16f2949f44947b7b49996143982c3b3 P bf5f1a218f4a4840b7fb5616d3b2ac35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:12.375973 23229 tablet_replica.cc:330] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca: stopping tablet replica
I20260809 04:35:12.376134 23229 raft_consensus.cc:2229] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:12.376271 23229 raft_consensus.cc:2258] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:12.377004 23229 ts_tablet_manager.cc:1905] T e16f2949f44947b7b49996143982c3b3 P 8875775338674f0d82511974d1dbb1ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 04:35:12.391062 22270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:38597 (error 108)
I20260809 04:35:12.391297 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 22582
I20260809 04:35:12.396466 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:12.395955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38597 (local address 127.0.0.1:56756)
0809 04:35:12.396249 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:12.396341 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
W20260809 04:35:12.396703 22270 catalog_manager.cc:4584] TS 86acf733d80244159f20abf759fb3603 (127.19.92.1:38597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e16f2949f44947b7b49996143982c3b3: Network error: Client connection negotiation failed: client connection to 127.19.92.1:38597: connect: Connection refused (error 111)
I20260809 04:35:12.449986 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:12.449655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38597 (local address 127.0.0.1:56772)
0809 04:35:12.449820 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:12.449897 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 04:35:12.450758 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:12.450560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38597 (local address 127.0.0.1:56774)
0809 04:35:12.450652 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:12.450701 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 04:35:12.467617 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 22783
I20260809 04:35:12.470928 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:12.470601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:37613 (local address 127.0.0.1:52052)
0809 04:35:12.470790 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:12.470837 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.2:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260809 04:35:12.484815 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 22245
2026-08-09T04:35:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6847 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:14 dist-test-slave-rmhg krb5kdc[23236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:14 dist-test-slave-rmhg krb5kdc[23236](info): set up 2 sockets
Aug 09 04:35:14 dist-test-slave-rmhg krb5kdc[23236](info): commencing operation
krb5kdc: starting...
W20260809 04:35:15.157138 19824 mini_kdc.cc:121] Time spent starting KDC: real 2.645s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:15 dist-test-slave-rmhg krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:15.281734 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:42815
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:42815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:15.398188 23254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:15.398468 23254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:15.398509 23254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:15.402470 23254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:15.402534 23254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:15.402552 23254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:15.402567 23254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:15.402580 23254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:15.407986 23254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:42815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:42815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:15.409134 23254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:15.410233 23254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:15.418808 23266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:15.418810 23261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:15.418833 23262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:15.418861 23254 server_base.cc:1034] running on GCE node
I20260809 04:35:15.419634 23254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:15.420769 23254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:15.421931 23254 hybrid_clock.cc:648] HybridClock initialized: now 1786250115421914 us; error 37 us; skew 500 ppm
Aug 09 04:35:15 dist-test-slave-rmhg krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:15.424737 23254 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:15.425734 23254 webserver.cc:460] Webserver started at http://127.19.92.62:36831/ using document root <none> and password file <none>
I20260809 04:35:15.426312 23254 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:15.426378 23254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:15.426559 23254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:15.428718 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "45e03996e49749fbafc7d1209b60f5eb"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "7b777de8f6a679139862470515480a90"
server_key_iv: "8f854517bc6e256da270f115f77f5691"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.429330 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "45e03996e49749fbafc7d1209b60f5eb"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "7b777de8f6a679139862470515480a90"
server_key_iv: "8f854517bc6e256da270f115f77f5691"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.433188 23254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:35:15.435889 23277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:15.437266 23254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:15.437386 23254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "45e03996e49749fbafc7d1209b60f5eb"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "7b777de8f6a679139862470515480a90"
server_key_iv: "8f854517bc6e256da270f115f77f5691"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.437486 23254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:15.466682 23254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:15.469694 23254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:15.469867 23254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:15.477025 23254 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:42815
I20260809 04:35:15.477063 23349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:42815 every 8 connection(s)
I20260809 04:35:15.478080 23254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:15.478775 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 23254
I20260809 04:35:15.478880 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:15.479143 19824 external_mini_cluster.cc:1428] Setting key 515d57c2dc8c5339b2486d2f3f6220ba
I20260809 04:35:15.481276 23350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:15.488461 23350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb: Bootstrap starting.
Aug 09 04:35:15 dist-test-slave-rmhg krb5kdc[23236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:15.491307 23350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:15.492188 23350 log.cc:824] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:15.494478 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:15.480524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.19.92.62:42815)
0809 04:35:15.480965 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:15.480986 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:15.481144 (+ 158us) server_negotiation.cc:407] Connection header received
0809 04:35:15.482106 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:15.482130 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:15.482558 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:15.482965 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:15.483647 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:15.485238 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:15.486091 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:15.486807 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:15.486973 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:15.489410 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:15.489417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:15.489433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:15.489468 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:15.491313 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:15.491767 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:15.491773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:15.491779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:15.491926 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:15.492157 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:15.492160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:15.492162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:15.492709 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:15.492856 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:15.493201 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 04:35:15.493518 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":111,"threads_started":1}
I20260809 04:35:15.494695 23350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb: No bootstrap required, opened a new log
I20260809 04:35:15.498035 23350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e03996e49749fbafc7d1209b60f5eb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 42815 } }
I20260809 04:35:15.498276 23350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:15.498332 23350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e03996e49749fbafc7d1209b60f5eb, State: Initialized, Role: FOLLOWER
I20260809 04:35:15.498865 23350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e03996e49749fbafc7d1209b60f5eb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 42815 } }
I20260809 04:35:15.499002 23350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:15.499065 23350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:15.499171 23350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:15.500077 23350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e03996e49749fbafc7d1209b60f5eb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 42815 } }
I20260809 04:35:15.500427 23350 leader_election.cc:302] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45e03996e49749fbafc7d1209b60f5eb; no voters:
I20260809 04:35:15.500738 23350 leader_election.cc:288] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:15.500954 23357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:15.501250 23357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [term 1 LEADER]: Becoming Leader. State: Replica: 45e03996e49749fbafc7d1209b60f5eb, State: Running, Role: LEADER
I20260809 04:35:15.501571 23357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e03996e49749fbafc7d1209b60f5eb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 42815 } }
I20260809 04:35:15.502012 23350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:15.503239 23360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e03996e49749fbafc7d1209b60f5eb. Latest consensus state: current_term: 1 leader_uuid: "45e03996e49749fbafc7d1209b60f5eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e03996e49749fbafc7d1209b60f5eb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 42815 } } }
I20260809 04:35:15.503384 23360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:15.503750 23358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45e03996e49749fbafc7d1209b60f5eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e03996e49749fbafc7d1209b60f5eb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 42815 } } }
I20260809 04:35:15.503854 23358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:15.503832 23368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:15.506919 23368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:15.512569 23368 catalog_manager.cc:1347] Generated new cluster ID: fd63c9a0f1444aaca1e0cd47dab2ac67
I20260809 04:35:15.512651 23368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:15.548957 23368 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:15.550001 23368 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:15.556851 23368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45e03996e49749fbafc7d1209b60f5eb: Generated new TSK 0
I20260809 04:35:15.557574 23368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:15.589608 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:42815
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:15.702247 23382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:15.702632 23382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:15.702710 23382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:15.706954 23382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:15.707029 23382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:15.707111 23382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:15.712589 23382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:15.713737 23382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:15.714802 23382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:15.724218 23393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:15.724188 23390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:15.724298 23382 server_base.cc:1034] running on GCE node
W20260809 04:35:15.724215 23389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:15.724845 23382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:15.725473 23382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:15.726634 23382 hybrid_clock.cc:648] HybridClock initialized: now 1786250115726617 us; error 26 us; skew 500 ppm
Aug 09 04:35:15 dist-test-slave-rmhg krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:15.729276 23382 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:15.730124 23382 webserver.cc:460] Webserver started at http://127.19.92.1:34851/ using document root <none> and password file <none>
I20260809 04:35:15.730633 23382 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:15.730686 23382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:15.730852 23382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:15.732792 23382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "00d05508becc4101a029f32f1f64f793"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "e04bec536a82c2e1ada7692be6d917d2"
server_key_iv: "e4369ebfedc521d271d3710e2f40c2ec"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.733332 23382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "00d05508becc4101a029f32f1f64f793"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "e04bec536a82c2e1ada7692be6d917d2"
server_key_iv: "e4369ebfedc521d271d3710e2f40c2ec"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.736792 23382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:15.739176 23406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:15.740351 23382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:15.740468 23382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "00d05508becc4101a029f32f1f64f793"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "e04bec536a82c2e1ada7692be6d917d2"
server_key_iv: "e4369ebfedc521d271d3710e2f40c2ec"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.740583 23382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:15.758610 23382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:15.761291 23382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:15.761494 23382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:15.762074 23382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:15.762920 23382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:15.762977 23382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:15.763032 23382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:15.763059 23382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:15.771474 23567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:42173 every 8 connection(s)
I20260809 04:35:15.771486 23382 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:42173
I20260809 04:35:15.772397 23382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:15.777112 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 23382
I20260809 04:35:15.777235 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:15.777478 19824 external_mini_cluster.cc:1428] Setting key ca61c67940a8e8cb878d4301ccf33df8
Aug 09 04:35:15 dist-test-slave-rmhg krb5kdc[23236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:15.786442 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:15.774147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:44107 (local address 127.19.92.62:42815)
0809 04:35:15.774295 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:15.774300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:15.775134 (+ 834us) server_negotiation.cc:407] Connection header received
0809 04:35:15.776377 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:15.776379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:15.776426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:15.776499 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:15.777940 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:15.778693 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:15.779928 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:15.780182 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:15.780275 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:15.782900 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:15.782904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:15.782906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:15.782933 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:15.784224 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:15.784777 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:15.784780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:15.784782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:15.784868 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:15.785240 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:15.785243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:15.785245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:15.785448 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:15.785539 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:15.786117 (+ 578us) server_negotiation.cc:299] Negotiation successful
0809 04:35:15.786268 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:35:15.787271 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:15.774544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:42815 (local address 127.19.92.1:44107)
0809 04:35:15.774984 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:15.775028 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:15.776179 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:15.776742 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:15.776750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:15.777134 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:15.777783 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:15.777798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:15.778810 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:15.778814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:15.779806 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:15.779817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:15.780334 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:15.780339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:15.780506 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:15.781107 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:15.781123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:15.782703 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:15.784347 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:15.784354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:15.784375 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:15.784671 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:15.784956 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:15.784959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:15.784962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:15.785117 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:15.785555 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:15.785563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:15.785855 (+ 292us) client_negotiation.cc:769] Sending connection context
0809 04:35:15.786031 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 04:35:15.786395 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:35:15.788653 23568 heartbeater.cc:347] Connected to a master server at 127.19.92.62:42815
I20260809 04:35:15.788890 23568 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:15.789458 23568 heartbeater.cc:511] Master 127.19.92.62:42815 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:35:15.790949 23300 ts_manager.cc:194] Registered new tserver with Master: 00d05508becc4101a029f32f1f64f793 (127.19.92.1:42173)
I20260809 04:35:15.792060 23300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:44107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:15.804919 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:42815
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:15.914764 23575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:15.915025 23575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:15.915091 23575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:15.919131 23575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:15.919188 23575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:15.919267 23575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:15.924365 23575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:15.925427 23575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:15.926409 23575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:15.935591 23585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:15.935618 23581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:15.935783 23575 server_base.cc:1034] running on GCE node
W20260809 04:35:15.935890 23582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:15.936264 23575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:15.936872 23575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:15.938028 23575 hybrid_clock.cc:648] HybridClock initialized: now 1786250115937998 us; error 38 us; skew 500 ppm
Aug 09 04:35:15 dist-test-slave-rmhg krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:15.942387 23575 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:15.943585 23575 webserver.cc:460] Webserver started at http://127.19.92.2:39833/ using document root <none> and password file <none>
I20260809 04:35:15.944271 23575 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:15.944351 23575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:15.944617 23575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:15.946907 23575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "c7299b81042b40ecae7a723cada25eea"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "e796974f55bfb5f5cecf2994b877e26b"
server_key_iv: "6f7484b325c22952731f3cf4f07799e1"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.947460 23575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "c7299b81042b40ecae7a723cada25eea"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "e796974f55bfb5f5cecf2994b877e26b"
server_key_iv: "6f7484b325c22952731f3cf4f07799e1"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.951172 23575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 04:35:15.953524 23593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:15.954705 23575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:15.954810 23575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "c7299b81042b40ecae7a723cada25eea"
format_stamp: "Formatted at 2026-08-09 04:35:15 on dist-test-slave-rmhg"
server_key: "e796974f55bfb5f5cecf2994b877e26b"
server_key_iv: "6f7484b325c22952731f3cf4f07799e1"
server_key_version: "encryptionkey@0"
I20260809 04:35:15.954896 23575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:16.012883 23575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:16.016237 23575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:16.016414 23575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:16.017023 23575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:16.018007 23575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:16.018059 23575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:16.018112 23575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:16.018136 23575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:16.026949 23575 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:41911
I20260809 04:35:16.026969 23761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:41911 every 8 connection(s)
I20260809 04:35:16.027853 23575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:16.031672 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 23575
I20260809 04:35:16.031761 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:16.032034 19824 external_mini_cluster.cc:1428] Setting key cdbcbd657f959fdfe4e503be925dc841
Aug 09 04:35:16 dist-test-slave-rmhg krb5kdc[23236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:16.041734 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:16.029659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:55345 (local address 127.19.92.62:42815)
0809 04:35:16.029806 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:16.029810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:16.030700 (+ 890us) server_negotiation.cc:407] Connection header received
0809 04:35:16.031966 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:16.031970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:16.032027 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:16.032132 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:16.033463 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:16.034185 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:16.035306 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:16.035581 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:16.035661 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:16.038177 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:16.038181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:16.038183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:16.038206 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:16.039554 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:16.040119 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:16.040121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:16.040122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:16.040192 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:16.040564 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:16.040566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:16.040568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:16.040739 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:16.040814 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:16.041506 (+ 692us) server_negotiation.cc:299] Negotiation successful
0809 04:35:16.041613 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:35:16.042809 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:16.030097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:42815 (local address 127.19.92.2:55345)
0809 04:35:16.030545 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:16.030589 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:16.031741 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:16.032290 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:16.032298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:16.032727 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:16.033310 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:16.033331 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:16.034293 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:16.034296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:16.035184 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:16.035193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:16.035720 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:16.035723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:16.035922 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:16.036528 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:16.036549 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:16.038023 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:16.039672 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:16.039678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:16.039694 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:16.040032 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:16.040267 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:16.040270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:16.040272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:16.040472 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:16.040838 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:16.040846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:16.041182 (+ 336us) client_negotiation.cc:769] Sending connection context
0809 04:35:16.041403 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 04:35:16.041741 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260809 04:35:16.043926 23763 heartbeater.cc:347] Connected to a master server at 127.19.92.62:42815
I20260809 04:35:16.044168 23763 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:16.044652 23763 heartbeater.cc:511] Master 127.19.92.62:42815 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:35:16.045814 23300 ts_manager.cc:194] Registered new tserver with Master: c7299b81042b40ecae7a723cada25eea (127.19.92.2:41911)
I20260809 04:35:16.046380 23300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:55345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:16.060815 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:42815
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:16.175025 23771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:16.175302 23771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:16.175346 23771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:16.179287 23771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:16.179347 23771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:16.179402 23771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:16.184742 23771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:16.185895 23771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:16.186905 23771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:16.196602 23780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:16.196768 23771 server_base.cc:1034] running on GCE node
W20260809 04:35:16.196550 23784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:16.196555 23781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:16.197376 23771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:16.198024 23771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:16.199219 23771 hybrid_clock.cc:648] HybridClock initialized: now 1786250116199198 us; error 29 us; skew 500 ppm
Aug 09 04:35:16 dist-test-slave-rmhg krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250116, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:16.201974 23771 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:16.202982 23771 webserver.cc:460] Webserver started at http://127.19.92.3:38015/ using document root <none> and password file <none>
I20260809 04:35:16.203579 23771 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:16.203656 23771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:16.203863 23771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:16.206029 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "6d54e3d486714b7796f06bc989d325b7"
format_stamp: "Formatted at 2026-08-09 04:35:16 on dist-test-slave-rmhg"
server_key: "e3ffc88574c89e9ec88832c20b1a75cd"
server_key_iv: "e890c650bfe5004df3eb6ac61e83a969"
server_key_version: "encryptionkey@0"
I20260809 04:35:16.206590 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "6d54e3d486714b7796f06bc989d325b7"
format_stamp: "Formatted at 2026-08-09 04:35:16 on dist-test-slave-rmhg"
server_key: "e3ffc88574c89e9ec88832c20b1a75cd"
server_key_iv: "e890c650bfe5004df3eb6ac61e83a969"
server_key_version: "encryptionkey@0"
I20260809 04:35:16.210342 23771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 04:35:16.212819 23792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:16.214188 23771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:35:16.214323 23771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "6d54e3d486714b7796f06bc989d325b7"
format_stamp: "Formatted at 2026-08-09 04:35:16 on dist-test-slave-rmhg"
server_key: "e3ffc88574c89e9ec88832c20b1a75cd"
server_key_iv: "e890c650bfe5004df3eb6ac61e83a969"
server_key_version: "encryptionkey@0"
I20260809 04:35:16.214435 23771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:16.230279 23771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:16.233589 23771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:16.233789 23771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:16.234418 23771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:16.235291 23771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:16.235342 23771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:16.235397 23771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:16.235425 23771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:16.244684 23771 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:46751
I20260809 04:35:16.244741 23957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:46751 every 8 connection(s)
I20260809 04:35:16.245720 23771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:16.247998 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 23771
I20260809 04:35:16.248100 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:16.248414 19824 external_mini_cluster.cc:1428] Setting key c9d5e2af5ee2b4b4e2a218e821305fe7
Aug 09 04:35:16 dist-test-slave-rmhg krb5kdc[23236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250116, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:16.259083 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:16.247504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:40803 (local address 127.19.92.62:42815)
0809 04:35:16.247634 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:16.247638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:16.248641 (+ 1003us) server_negotiation.cc:407] Connection header received
0809 04:35:16.250055 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:16.250059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:16.250115 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:16.250197 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:16.251394 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:16.251914 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:16.253074 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:16.253358 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:16.253438 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:16.255819 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:16.255823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:16.255825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:16.255850 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:16.257226 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:16.257732 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:16.257736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:16.257738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:16.257825 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:16.258128 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:16.258131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:16.258132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:16.258302 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:16.258373 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:16.258854 (+ 481us) server_negotiation.cc:299] Negotiation successful
0809 04:35:16.258960 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 04:35:16.259914 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:16.247877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:42815 (local address 127.19.92.3:40803)
0809 04:35:16.248452 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:16.248525 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:16.249859 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:16.250389 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:16.250398 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:16.250787 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:16.251276 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:16.251289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:16.252001 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:16.252004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:16.252956 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:16.252966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:16.253497 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:16.253500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:16.253658 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:16.254194 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:16.254210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:16.255660 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:16.257322 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:16.257327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:16.257339 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:16.257623 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:16.257908 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:16.257910 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:16.257912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:16.258058 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:16.258372 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:16.258376 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:16.258642 (+ 266us) client_negotiation.cc:769] Sending connection context
0809 04:35:16.258809 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 04:35:16.259103 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":147,"threads_started":1}
I20260809 04:35:16.261073 23958 heartbeater.cc:347] Connected to a master server at 127.19.92.62:42815
I20260809 04:35:16.261338 23958 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:16.261884 23958 heartbeater.cc:511] Master 127.19.92.62:42815 requested a full tablet report, sending...
I20260809 04:35:16.263058 23300 ts_manager.cc:194] Registered new tserver with Master: 6d54e3d486714b7796f06bc989d325b7 (127.19.92.3:46751)
I20260809 04:35:16.263654 23300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:40803
I20260809 04:35:16.274598 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 04:35:16.296084 23353 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:16.294536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.19.92.62:42815)
0809 04:35:16.294676 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:16.294681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:16.294804 (+ 123us) server_negotiation.cc:407] Connection header received
0809 04:35:16.294937 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:16.294939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:16.294984 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:16.295054 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:16.295929 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42146: BlockingRecv error: recv got EOF from 127.0.0.1:42146 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:35:16.297302 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 23382
I20260809 04:35:16.325145 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 23575
I20260809 04:35:16.335532 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 23771
I20260809 04:35:16.344393 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 23254
2026-08-09T04:35:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3861 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:16 dist-test-slave-rmhg krb5kdc[23982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:16 dist-test-slave-rmhg krb5kdc[23982](info): set up 2 sockets
Aug 09 04:35:16 dist-test-slave-rmhg krb5kdc[23982](info): commencing operation
krb5kdc: starting...
W20260809 04:35:17.476938 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:17 dist-test-slave-rmhg krb5kdc[23982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:17.606292 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:39489
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:17.719777 24005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:17.720074 24005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:17.720134 24005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:17.724225 24005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:17.724282 24005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:17.724309 24005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:17.724330 24005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:17.724350 24005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:17.729892 24005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:39489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:17.731078 24005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:17.732124 24005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:17.740478 24014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:17.740497 24005 server_base.cc:1034] running on GCE node
W20260809 04:35:17.740444 24022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:17.740444 24015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:17.741299 24005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:17.742415 24005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:17.743608 24005 hybrid_clock.cc:648] HybridClock initialized: now 1786250117743589 us; error 31 us; skew 500 ppm
Aug 09 04:35:17 dist-test-slave-rmhg krb5kdc[23982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250117, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:17.746608 24005 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:17.747586 24005 webserver.cc:460] Webserver started at http://127.19.92.62:39005/ using document root <none> and password file <none>
I20260809 04:35:17.748128 24005 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:17.748188 24005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:17.748364 24005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:17.750507 24005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "c91a7b58510a47198328d8a52866f5f9"
format_stamp: "Formatted at 2026-08-09 04:35:17 on dist-test-slave-rmhg"
server_key: "c5e6c56fbbbd2464d53fed363b446749"
server_key_iv: "4d4211a57e20efa731c245770445ed19"
server_key_version: "encryptionkey@0"
I20260809 04:35:17.751118 24005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "c91a7b58510a47198328d8a52866f5f9"
format_stamp: "Formatted at 2026-08-09 04:35:17 on dist-test-slave-rmhg"
server_key: "c5e6c56fbbbd2464d53fed363b446749"
server_key_iv: "4d4211a57e20efa731c245770445ed19"
server_key_version: "encryptionkey@0"
I20260809 04:35:17.754928 24005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:35:17.757386 24033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:17.758646 24005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:17.758754 24005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "c91a7b58510a47198328d8a52866f5f9"
format_stamp: "Formatted at 2026-08-09 04:35:17 on dist-test-slave-rmhg"
server_key: "c5e6c56fbbbd2464d53fed363b446749"
server_key_iv: "4d4211a57e20efa731c245770445ed19"
server_key_version: "encryptionkey@0"
I20260809 04:35:17.758855 24005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:17.780402 24005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:17.783519 24005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:17.783711 24005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:17.790848 24005 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:39489
I20260809 04:35:17.791108 24107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:39489 every 8 connection(s)
I20260809 04:35:17.791841 24005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:17.793780 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 24005
I20260809 04:35:17.793886 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:17.794145 19824 external_mini_cluster.cc:1428] Setting key efccef4591970e4eff15c71c116e4d63
I20260809 04:35:17.795112 24108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:17.802111 24108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9: Bootstrap starting.
Aug 09 04:35:17 dist-test-slave-rmhg krb5kdc[23982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:17.804494 24108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:17.805408 24108 log.cc:824] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:17.807619 24108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9: No bootstrap required, opened a new log
I20260809 04:35:17.809324 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:17.795472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.19.92.62:39489)
0809 04:35:17.795911 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:17.795940 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:17.795984 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:35:17.796815 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:17.796840 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:17.797305 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:17.797678 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:17.798382 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:17.799863 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:17.800793 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:17.801517 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:17.801683 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:17.804055 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:17.804062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:17.804078 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:17.804110 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:17.806017 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:17.806433 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:17.806439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:17.806444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:17.806580 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:17.806837 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:17.806840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:17.806842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:17.807354 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:17.807535 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:17.808137 (+ 602us) server_negotiation.cc:299] Negotiation successful
0809 04:35:17.808370 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260809 04:35:17.810714 24108 raft_consensus.cc:348] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91a7b58510a47198328d8a52866f5f9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39489 } }
I20260809 04:35:17.810931 24108 raft_consensus.cc:374] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:17.810982 24108 raft_consensus.cc:729] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91a7b58510a47198328d8a52866f5f9, State: Initialized, Role: FOLLOWER
I20260809 04:35:17.811410 24108 consensus_queue.cc:260] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91a7b58510a47198328d8a52866f5f9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39489 } }
I20260809 04:35:17.811545 24108 raft_consensus.cc:388] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:17.811607 24108 raft_consensus.cc:482] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:17.811697 24108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:17.812096 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:35:17.812579 24108 raft_consensus.cc:504] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91a7b58510a47198328d8a52866f5f9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39489 } }
I20260809 04:35:17.812885 24108 leader_election.cc:302] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c91a7b58510a47198328d8a52866f5f9; no voters:
I20260809 04:35:17.813215 24108 leader_election.cc:288] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:17.813303 24118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:17.813491 24118 raft_consensus.cc:686] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [term 1 LEADER]: Becoming Leader. State: Replica: c91a7b58510a47198328d8a52866f5f9, State: Running, Role: LEADER
I20260809 04:35:17.813786 24118 consensus_queue.cc:237] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91a7b58510a47198328d8a52866f5f9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39489 } }
I20260809 04:35:17.814502 24108 sys_catalog.cc:564] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:17.814963 24118 sys_catalog.cc:455] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c91a7b58510a47198328d8a52866f5f9. Latest consensus state: current_term: 1 leader_uuid: "c91a7b58510a47198328d8a52866f5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91a7b58510a47198328d8a52866f5f9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39489 } } }
I20260809 04:35:17.815074 24118 sys_catalog.cc:458] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:17.815361 24119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c91a7b58510a47198328d8a52866f5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91a7b58510a47198328d8a52866f5f9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39489 } } }
I20260809 04:35:17.815452 24119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9 [sys.catalog]: This master's current role is: LEADER
W20260809 04:35:17.819242 24141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:35:17.819324 24141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:35:17.819381 24126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:17.820211 24126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:17.826701 24126 catalog_manager.cc:1347] Generated new cluster ID: 86f7f700ea184fc8814798ac58ed04ff
I20260809 04:35:17.826776 24126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:17.832499 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:17.826606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.19.92.62:39489)
0809 04:35:17.826738 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:17.826743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:17.827187 (+ 444us) server_negotiation.cc:407] Connection header received
0809 04:35:17.827345 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:17.827349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:17.827400 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:17.827455 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:17.828242 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:17.829046 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:17.830241 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:17.830575 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:17.830663 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:17.831555 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:17.831559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:17.831562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:17.832075 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:17.832108 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:17.832126 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:17.832207 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:17.832304 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 04:35:17.832361 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:35:17.843568 24126 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:17.844440 24126 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:17.852631 24126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c91a7b58510a47198328d8a52866f5f9: Generated new TSK 0
I20260809 04:35:17.853390 24126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:35:17.861794 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 24005
2026-08-09T04:35:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1516 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:17 dist-test-slave-rmhg krb5kdc[24156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:17 dist-test-slave-rmhg krb5kdc[24156](info): set up 2 sockets
Aug 09 04:35:17 dist-test-slave-rmhg krb5kdc[24156](info): commencing operation
krb5kdc: starting...
W20260809 04:35:18.992985 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:19 dist-test-slave-rmhg krb5kdc[24156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:19.121928 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:37287
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:19.234557 24179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:19.234848 24179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:19.234906 24179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:19.238915 24179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:19.238972 24179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:19.238998 24179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:19.239018 24179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:19.239038 24179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:19.244215 24179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:37287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:19.245293 24179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:19.246290 24179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:19.254551 24190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:19.254565 24186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:19.254568 24185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:19.254808 24179 server_base.cc:1034] running on GCE node
I20260809 04:35:19.255439 24179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:19.256500 24179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:19.257660 24179 hybrid_clock.cc:648] HybridClock initialized: now 1786250119257638 us; error 34 us; skew 500 ppm
Aug 09 04:35:19 dist-test-slave-rmhg krb5kdc[24156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:19.263293 24179 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:19.264303 24179 webserver.cc:460] Webserver started at http://127.19.92.62:43713/ using document root <none> and password file <none>
I20260809 04:35:19.264902 24179 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:19.264973 24179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:19.265226 24179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:19.267447 24179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "55a0551713d846f09eaa652bab3bbd74"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "a0b7e94822524bf0e870756ddb84d4f1"
server_key_iv: "ed3c5a0e1ebf02759709622d6da644e0"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.268049 24179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "55a0551713d846f09eaa652bab3bbd74"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "a0b7e94822524bf0e870756ddb84d4f1"
server_key_iv: "ed3c5a0e1ebf02759709622d6da644e0"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.271957 24179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:35:19.274539 24201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:19.275791 24179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:19.275924 24179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "55a0551713d846f09eaa652bab3bbd74"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "a0b7e94822524bf0e870756ddb84d4f1"
server_key_iv: "ed3c5a0e1ebf02759709622d6da644e0"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.276021 24179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:19.293776 24179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:19.296732 24179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:19.296908 24179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:19.303535 24179 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:37287
I20260809 04:35:19.303813 24284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:37287 every 8 connection(s)
I20260809 04:35:19.304553 24179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:19.307529 24287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:19.309520 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 24179
I20260809 04:35:19.309646 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:19.309892 19824 external_mini_cluster.cc:1428] Setting key 8a9dc362087861dac25a5f47f1aefedb
I20260809 04:35:19.314656 24287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74: Bootstrap starting.
I20260809 04:35:19.317569 24287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:19.318398 24287 log.cc:824] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:35:19 dist-test-slave-rmhg krb5kdc[24156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:19.320472 24287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74: No bootstrap required, opened a new log
I20260809 04:35:19.325217 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:19.311380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.19.92.62:37287)
0809 04:35:19.311829 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:19.311850 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:19.311894 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:35:19.312635 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:19.312662 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:19.313101 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:19.313524 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:19.314163 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.315662 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.316877 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.317627 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.317787 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:19.320107 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:19.320113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:19.320129 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:19.320162 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:19.322057 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:19.322471 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:19.322478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:19.322484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:19.322631 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:19.322862 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:19.322865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:19.322867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:19.323395 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:19.323587 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:19.323808 (+ 221us) server_negotiation.cc:299] Negotiation successful
0809 04:35:19.324148 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260809 04:35:19.325086 24287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a0551713d846f09eaa652bab3bbd74" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37287 } }
I20260809 04:35:19.325378 24287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:19.325438 24287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a0551713d846f09eaa652bab3bbd74, State: Initialized, Role: FOLLOWER
I20260809 04:35:19.325843 24287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a0551713d846f09eaa652bab3bbd74" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37287 } }
I20260809 04:35:19.325986 24287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:19.326051 24287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:19.326151 24287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:19.327283 24287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a0551713d846f09eaa652bab3bbd74" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37287 } }
I20260809 04:35:19.327651 24287 leader_election.cc:302] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a0551713d846f09eaa652bab3bbd74; no voters:
I20260809 04:35:19.328033 24287 leader_election.cc:288] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:19.328122 24298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:19.328351 24298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [term 1 LEADER]: Becoming Leader. State: Replica: 55a0551713d846f09eaa652bab3bbd74, State: Running, Role: LEADER
I20260809 04:35:19.328651 24298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a0551713d846f09eaa652bab3bbd74" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37287 } }
I20260809 04:35:19.329237 24287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:19.330214 24299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55a0551713d846f09eaa652bab3bbd74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a0551713d846f09eaa652bab3bbd74" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37287 } } }
I20260809 04:35:19.330336 24299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:19.330652 24301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55a0551713d846f09eaa652bab3bbd74. Latest consensus state: current_term: 1 leader_uuid: "55a0551713d846f09eaa652bab3bbd74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a0551713d846f09eaa652bab3bbd74" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37287 } } }
I20260809 04:35:19.330818 24301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:19.331481 24311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:19.334076 24311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:19.339327 24311 catalog_manager.cc:1347] Generated new cluster ID: 096710adc5a54be7af7f668051d6c660
I20260809 04:35:19.339391 24311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:19.348127 24311 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:19.349507 24311 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:19.359283 24311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55a0551713d846f09eaa652bab3bbd74: Generated new TSK 0
I20260809 04:35:19.359982 24311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:19.390370 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37287
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:19.502674 24331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:19.502988 24331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:19.503053 24331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:19.507444 24331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:19.507560 24331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:19.507647 24331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:19.513433 24331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:19.514603 24331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:19.515687 24331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:19.525398 24337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:19.525440 24340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:19.525641 24331 server_base.cc:1034] running on GCE node
W20260809 04:35:19.525421 24344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:19.526165 24331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:19.526794 24331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:19.527971 24331 hybrid_clock.cc:648] HybridClock initialized: now 1786250119527949 us; error 35 us; skew 500 ppm
Aug 09 04:35:19 dist-test-slave-rmhg krb5kdc[24156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:19.530951 24331 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:19.531848 24331 webserver.cc:460] Webserver started at http://127.19.92.1:34875/ using document root <none> and password file <none>
I20260809 04:35:19.532397 24331 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:19.532498 24331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:19.532716 24331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:19.534835 24331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "9e160d09efb640d39551de875621900f"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "c643f1015ea9c3c2161a65ba1442f192"
server_key_iv: "2558685ce066d4df4d73d75228271c7f"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.535398 24331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "9e160d09efb640d39551de875621900f"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "c643f1015ea9c3c2161a65ba1442f192"
server_key_iv: "2558685ce066d4df4d73d75228271c7f"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.539156 24331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 04:35:19.541675 24353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:19.543063 24331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:19.543191 24331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "9e160d09efb640d39551de875621900f"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "c643f1015ea9c3c2161a65ba1442f192"
server_key_iv: "2558685ce066d4df4d73d75228271c7f"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.543299 24331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:19.554898 24331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:19.557835 24331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:19.558030 24331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:19.558641 24331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:19.559497 24331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:19.559547 24331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:19.559613 24331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:19.559640 24331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:19.568607 24331 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:39335
I20260809 04:35:19.568624 24524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:39335 every 8 connection(s)
I20260809 04:35:19.569576 24331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:19.577998 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 24331
I20260809 04:35:19.578127 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:19.578392 19824 external_mini_cluster.cc:1428] Setting key ec69db2b7483e9e83c304f903e68dbb8
Aug 09 04:35:19 dist-test-slave-rmhg krb5kdc[24156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:19.583492 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:19.571401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:37377 (local address 127.19.92.62:37287)
0809 04:35:19.571545 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:19.571549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:19.572589 (+ 1040us) server_negotiation.cc:407] Connection header received
0809 04:35:19.573981 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:19.573985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:19.574045 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:19.574125 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:19.575340 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.575884 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.577046 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.577320 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.577418 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:19.579985 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:19.579989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:19.579992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:19.580046 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:19.581516 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:19.582008 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:19.582012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:19.582013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:19.582096 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:19.582411 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:19.582414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:19.582416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:19.582607 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:19.582695 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:19.583197 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 04:35:19.583326 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:35:19.584400 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:19.571916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37287 (local address 127.19.92.1:37377)
0809 04:35:19.572422 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:19.572474 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:19.573758 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:19.574294 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:19.574301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:19.574673 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:19.575181 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:19.575194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.576009 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.576012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:19.576915 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:19.576925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.577466 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.577470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:19.577642 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:19.578305 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:19.578322 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:19.579820 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:19.581633 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:19.581638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:19.581650 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:19.581909 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:19.582183 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:19.582186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:19.582187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:19.582322 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:19.582707 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:19.582714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:19.582958 (+ 244us) client_negotiation.cc:769] Sending connection context
0809 04:35:19.583135 (+ 177us) client_negotiation.cc:240] Negotiation successful
0809 04:35:19.583462 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":147,"threads_started":1}
I20260809 04:35:19.585652 24525 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37287
I20260809 04:35:19.585901 24525 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:19.586421 24525 heartbeater.cc:511] Master 127.19.92.62:37287 requested a full tablet report, sending...
I20260809 04:35:19.588030 24226 ts_manager.cc:194] Registered new tserver with Master: 9e160d09efb640d39551de875621900f (127.19.92.1:39335)
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:35:19.589299 24226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:37377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:19.607828 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37287
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:19.722803 24531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:19.723084 24531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:19.723127 24531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:19.727288 24531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:19.727356 24531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:19.727422 24531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:19.732955 24531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:19.734089 24531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:19.735175 24531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:19.744737 24537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:19.744782 24541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:19.744813 24531 server_base.cc:1034] running on GCE node
W20260809 04:35:19.745045 24539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:19.745492 24531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:19.746143 24531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:19.747314 24531 hybrid_clock.cc:648] HybridClock initialized: now 1786250119747278 us; error 52 us; skew 500 ppm
Aug 09 04:35:19 dist-test-slave-rmhg krb5kdc[24156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:19.750275 24531 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:19.751247 24531 webserver.cc:460] Webserver started at http://127.19.92.2:40197/ using document root <none> and password file <none>
I20260809 04:35:19.751812 24531 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:19.751883 24531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:19.752094 24531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:19.754647 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "8d807d8f54584198870ea38172e25d6c"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "cfe05a824b46c0e026fed85e3c3d715c"
server_key_iv: "26df67ee9060debd439cf3e632fd3cf7"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.755232 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "8d807d8f54584198870ea38172e25d6c"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "cfe05a824b46c0e026fed85e3c3d715c"
server_key_iv: "26df67ee9060debd439cf3e632fd3cf7"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.759203 24531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:35:19.761662 24549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:19.762912 24531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:19.763043 24531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "8d807d8f54584198870ea38172e25d6c"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "cfe05a824b46c0e026fed85e3c3d715c"
server_key_iv: "26df67ee9060debd439cf3e632fd3cf7"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.763139 24531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:19.776926 24531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:19.779976 24531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:19.780169 24531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:19.780763 24531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:19.781630 24531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:19.781679 24531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:19.781733 24531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:19.781761 24531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:19.790735 24531 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:41519
I20260809 04:35:19.790787 24712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:41519 every 8 connection(s)
I20260809 04:35:19.791685 24531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:19.795567 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 24531
I20260809 04:35:19.795701 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:19.795981 19824 external_mini_cluster.cc:1428] Setting key e5ca70a8616ceaca0cd4f27416175b76
Aug 09 04:35:19 dist-test-slave-rmhg krb5kdc[24156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:19.806854 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:19.793561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:60647 (local address 127.19.92.62:37287)
0809 04:35:19.793726 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:19.793730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:19.794598 (+ 868us) server_negotiation.cc:407] Connection header received
0809 04:35:19.796181 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:19.796184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:19.796237 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:19.796314 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:19.797779 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.798300 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.799840 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.800169 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.800273 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:19.803287 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:19.803290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:19.803292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:19.803318 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:19.804639 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:19.805269 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:19.805271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:19.805272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:19.805346 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:19.805713 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:19.805716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:19.805717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:19.805891 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:19.805965 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:19.806625 (+ 660us) server_negotiation.cc:299] Negotiation successful
0809 04:35:19.806730 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:35:19.807825 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:19.793975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37287 (local address 127.19.92.2:60647)
0809 04:35:19.794435 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:19.794481 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:19.795939 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:19.796515 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:19.796522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:19.796953 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:19.797613 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:19.797634 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.798426 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.798430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:19.799720 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:19.799732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:19.800327 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:19.800330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:19.800489 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:19.801123 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:19.801147 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:19.803108 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:19.804761 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:19.804768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:19.804785 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:19.805174 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:19.805430 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:19.805434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:19.805436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:19.805622 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:19.805993 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:19.806000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:19.806349 (+ 349us) client_negotiation.cc:769] Sending connection context
0809 04:35:19.806568 (+ 219us) client_negotiation.cc:240] Negotiation successful
0809 04:35:19.806928 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":131,"threads_started":1}
I20260809 04:35:19.808929 24713 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37287
I20260809 04:35:19.809178 24713 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:35:19.809751 24713 heartbeater.cc:511] Master 127.19.92.62:37287 requested a full tablet report, sending...
I20260809 04:35:19.810835 24226 ts_manager.cc:194] Registered new tserver with Master: 8d807d8f54584198870ea38172e25d6c (127.19.92.2:41519)
I20260809 04:35:19.811568 24226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:60647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:19.824488 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37287
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:19.940171 24722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:19.940442 24722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:19.940503 24722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:19.944712 24722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:19.944788 24722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:19.944872 24722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:19.950438 24722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:19.951640 24722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:19.952677 24722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:19.962423 24730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:19.962426 24729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:19.962479 24734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:19.962775 24722 server_base.cc:1034] running on GCE node
I20260809 04:35:19.963301 24722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:19.963924 24722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:19.965097 24722 hybrid_clock.cc:648] HybridClock initialized: now 1786250119965078 us; error 30 us; skew 500 ppm
Aug 09 04:35:19 dist-test-slave-rmhg krb5kdc[24156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:19.967880 24722 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:19.968788 24722 webserver.cc:460] Webserver started at http://127.19.92.3:40273/ using document root <none> and password file <none>
I20260809 04:35:19.969383 24722 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:19.969446 24722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:19.969657 24722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:19.971709 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "a3f7d1543f3d437d8201af8cf5b41d39"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "33a116e432bbd8a07413b3f4b6c07899"
server_key_iv: "89d648af96c947f7b292154092196715"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.972254 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "a3f7d1543f3d437d8201af8cf5b41d39"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "33a116e432bbd8a07413b3f4b6c07899"
server_key_iv: "89d648af96c947f7b292154092196715"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.975927 24722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 04:35:19.978320 24744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:19.979561 24722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:19.979671 24722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "a3f7d1543f3d437d8201af8cf5b41d39"
format_stamp: "Formatted at 2026-08-09 04:35:19 on dist-test-slave-rmhg"
server_key: "33a116e432bbd8a07413b3f4b6c07899"
server_key_iv: "89d648af96c947f7b292154092196715"
server_key_version: "encryptionkey@0"
I20260809 04:35:19.979763 24722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:20.009057 24722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:20.012370 24722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:20.012557 24722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:20.013199 24722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:20.014035 24722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:20.014102 24722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:20.014166 24722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:20.014197 24722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:20.022747 24722 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:42911
I20260809 04:35:20.022784 24910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:42911 every 8 connection(s)
I20260809 04:35:20.023643 24722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:20.032794 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 24722
I20260809 04:35:20.032898 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:20.033185 19824 external_mini_cluster.cc:1428] Setting key 198b3cce1891f28a5e3999de9cea52b3
Aug 09 04:35:20 dist-test-slave-rmhg krb5kdc[24156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250119, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:20.037117 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:20.025552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:48245 (local address 127.19.92.62:37287)
0809 04:35:20.025706 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:20.025711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:20.026637 (+ 926us) server_negotiation.cc:407] Connection header received
0809 04:35:20.027969 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:20.027972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:20.028025 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:20.028107 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:20.029353 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:20.029990 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:20.031186 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:20.031403 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:20.031482 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:20.033908 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:20.033912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:20.033914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:20.033941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:20.035480 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:20.035922 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:20.035925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:20.035926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:20.035995 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:20.036235 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:20.036237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:20.036238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:20.036396 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:20.036465 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:20.036893 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 04:35:20.036995 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:35:20.037969 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:20.025978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37287 (local address 127.19.92.3:48245)
0809 04:35:20.026466 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:20.026515 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:20.027759 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:20.028290 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:20.028297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:20.028693 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:20.029235 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:20.029250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:20.030103 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:20.030106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:20.031080 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:20.031090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:20.031519 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:20.031521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:20.031662 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:20.032197 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:20.032214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:20.033731 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:20.035587 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:20.035592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:20.035605 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:20.035853 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:20.036048 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:20.036049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:20.036051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:20.036177 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:20.036464 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:20.036468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:20.036704 (+ 236us) client_negotiation.cc:769] Sending connection context
0809 04:35:20.036857 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 04:35:20.037125 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260809 04:35:20.038986 24911 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37287
I20260809 04:35:20.039217 24911 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:20.039695 24911 heartbeater.cc:511] Master 127.19.92.62:37287 requested a full tablet report, sending...
I20260809 04:35:20.040659 24226 ts_manager.cc:194] Registered new tserver with Master: a3f7d1543f3d437d8201af8cf5b41d39 (127.19.92.3:42911)
I20260809 04:35:20.041301 24226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:48245
I20260809 04:35:20.046893 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:35:20 dist-test-slave-rmhg krb5kdc[24156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250120, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 04:35:20 dist-test-slave-rmhg krb5kdc[24156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250120, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:20.072832 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:20.064595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.19.92.62:37287)
0809 04:35:20.064720 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:20.064724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:20.064894 (+ 170us) server_negotiation.cc:407] Connection header received
0809 04:35:20.065010 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:20.065013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:20.065061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:20.065141 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:20.065915 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:20.066469 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:20.067385 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:20.067636 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:20.067732 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:20.070070 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:20.070073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:20.070075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:20.070102 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:20.071371 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:20.071779 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:20.071782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:20.071783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:20.071870 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:20.072106 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:20.072108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:20.072109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:20.072330 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:20.072441 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:20.072607 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 04:35:20.072706 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260809 04:35:20.073238 24226 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35388
I20260809 04:35:20.074357 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 24331
I20260809 04:35:20.083986 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 24531
I20260809 04:35:20.099450 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 24722
I20260809 04:35:20.116494 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 24179
2026-08-09T04:35:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2258 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:20 dist-test-slave-rmhg krb5kdc[24927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:20 dist-test-slave-rmhg krb5kdc[24927](info): set up 2 sockets
Aug 09 04:35:20 dist-test-slave-rmhg krb5kdc[24927](info): commencing operation
krb5kdc: starting...
W20260809 04:35:21.261241 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:21 dist-test-slave-rmhg krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:21.387790 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36747
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:21.500813 24946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:21.501118 24946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:21.501204 24946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:21.505257 24946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:21.505330 24946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:21.505357 24946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:21.505378 24946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:21.505396 24946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:21.511315 24946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:21.512495 24946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:21.513588 24946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:35:21.522338 24946 server_base.cc:1034] running on GCE node
W20260809 04:35:21.522258 24956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:21.522249 24955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:21.522369 24958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:21.523159 24946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:21.524263 24946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:21.525432 24946 hybrid_clock.cc:648] HybridClock initialized: now 1786250121525407 us; error 44 us; skew 500 ppm
Aug 09 04:35:21 dist-test-slave-rmhg krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250121, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:21.528182 24946 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:21.529132 24946 webserver.cc:460] Webserver started at http://127.19.92.62:37751/ using document root <none> and password file <none>
I20260809 04:35:21.529711 24946 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:21.529781 24946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:21.529986 24946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:21.532037 24946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "3b75293604c14ecda48d9d97e0db84da"
format_stamp: "Formatted at 2026-08-09 04:35:21 on dist-test-slave-rmhg"
server_key: "ac9cc4ccc25eca7532fde42f3d1119dd"
server_key_iv: "7c98c999f292dd98794059c115e3a5df"
server_key_version: "encryptionkey@0"
I20260809 04:35:21.532585 24946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "3b75293604c14ecda48d9d97e0db84da"
format_stamp: "Formatted at 2026-08-09 04:35:21 on dist-test-slave-rmhg"
server_key: "ac9cc4ccc25eca7532fde42f3d1119dd"
server_key_iv: "7c98c999f292dd98794059c115e3a5df"
server_key_version: "encryptionkey@0"
I20260809 04:35:21.536362 24946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260809 04:35:21.538787 24968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:21.540908 24946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:35:21.541050 24946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "3b75293604c14ecda48d9d97e0db84da"
format_stamp: "Formatted at 2026-08-09 04:35:21 on dist-test-slave-rmhg"
server_key: "ac9cc4ccc25eca7532fde42f3d1119dd"
server_key_iv: "7c98c999f292dd98794059c115e3a5df"
server_key_version: "encryptionkey@0"
I20260809 04:35:21.541145 24946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:21.569187 24946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:21.572295 24946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:21.572486 24946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:21.579770 24946 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:36747
I20260809 04:35:21.579805 25044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:36747 every 8 connection(s)
I20260809 04:35:21.580766 24946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:21.583665 25045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:21.585639 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 24946
I20260809 04:35:21.585757 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:21.586022 19824 external_mini_cluster.cc:1428] Setting key 86b6eee6e874e05f18d7ce05173b33f7
I20260809 04:35:21.590956 25045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da: Bootstrap starting.
I20260809 04:35:21.593448 25045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:21.594256 25045 log.cc:824] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:35:21 dist-test-slave-rmhg krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:21.596491 25045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da: No bootstrap required, opened a new log
I20260809 04:35:21.600018 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:21.587408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.19.92.62:36747)
0809 04:35:21.587845 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:21.587861 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:21.587902 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:35:21.588615 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:21.588636 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:21.589008 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:21.589398 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:21.590266 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:21.591325 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:21.592194 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:21.592686 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:21.592843 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:21.595522 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:21.595528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:21.595544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:21.595582 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:21.597213 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:21.597638 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:21.597643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:21.597647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:21.597762 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:21.598029 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:21.598031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:21.598033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:21.598449 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:21.598641 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:21.598929 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 04:35:21.599235 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260809 04:35:21.601023 25045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b75293604c14ecda48d9d97e0db84da" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36747 } }
I20260809 04:35:21.601297 25045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:21.601372 25045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b75293604c14ecda48d9d97e0db84da, State: Initialized, Role: FOLLOWER
I20260809 04:35:21.601820 25045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b75293604c14ecda48d9d97e0db84da" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36747 } }
I20260809 04:35:21.601969 25045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:21.602036 25045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:21.602133 25045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:21.603053 25045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b75293604c14ecda48d9d97e0db84da" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36747 } }
I20260809 04:35:21.603385 25045 leader_election.cc:302] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b75293604c14ecda48d9d97e0db84da; no voters:
I20260809 04:35:21.603696 25045 leader_election.cc:288] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:21.603802 25050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:21.604014 25050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [term 1 LEADER]: Becoming Leader. State: Replica: 3b75293604c14ecda48d9d97e0db84da, State: Running, Role: LEADER
I20260809 04:35:21.604354 25050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b75293604c14ecda48d9d97e0db84da" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36747 } }
I20260809 04:35:21.604871 25045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:21.605933 25052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b75293604c14ecda48d9d97e0db84da. Latest consensus state: current_term: 1 leader_uuid: "3b75293604c14ecda48d9d97e0db84da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b75293604c14ecda48d9d97e0db84da" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36747 } } }
I20260809 04:35:21.606065 25052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:21.606449 25061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:21.606405 25051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b75293604c14ecda48d9d97e0db84da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b75293604c14ecda48d9d97e0db84da" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36747 } } }
I20260809 04:35:21.606629 25051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:21.609982 25061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:21.615595 25061 catalog_manager.cc:1347] Generated new cluster ID: 37cb1f4c65484b9398b3e26fec12bf13
I20260809 04:35:21.615674 25061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:21.629334 25061 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:21.630607 25061 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:21.642414 25061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b75293604c14ecda48d9d97e0db84da: Generated new TSK 0
I20260809 04:35:21.643169 25061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:21.677043 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36747
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:21.793072 25078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:21.793403 25078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:21.793474 25078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:21.797703 25078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:21.797773 25078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:21.797855 25078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:21.803269 25078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:21.804354 25078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:21.805500 25078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:21.815055 25087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:21.815049 25092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:21.815049 25086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:21.815326 25078 server_base.cc:1034] running on GCE node
I20260809 04:35:21.815773 25078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:21.816392 25078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:21.817559 25078 hybrid_clock.cc:648] HybridClock initialized: now 1786250121817538 us; error 30 us; skew 500 ppm
Aug 09 04:35:21 dist-test-slave-rmhg krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250121, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:21.820256 25078 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:21.821216 25078 webserver.cc:460] Webserver started at http://127.19.92.1:33599/ using document root <none> and password file <none>
I20260809 04:35:21.821776 25078 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:21.821838 25078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:21.822014 25078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:21.824096 25078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "448dbe2c26f545c193068ad28565d57d"
format_stamp: "Formatted at 2026-08-09 04:35:21 on dist-test-slave-rmhg"
server_key: "9084490b06074587c3cfff77b6073e30"
server_key_iv: "98ff6f05f56b505ab0b947579205b187"
server_key_version: "encryptionkey@0"
I20260809 04:35:21.824683 25078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "448dbe2c26f545c193068ad28565d57d"
format_stamp: "Formatted at 2026-08-09 04:35:21 on dist-test-slave-rmhg"
server_key: "9084490b06074587c3cfff77b6073e30"
server_key_iv: "98ff6f05f56b505ab0b947579205b187"
server_key_version: "encryptionkey@0"
I20260809 04:35:21.828764 25078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:35:21.831553 25103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:21.832902 25078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:21.833029 25078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "448dbe2c26f545c193068ad28565d57d"
format_stamp: "Formatted at 2026-08-09 04:35:21 on dist-test-slave-rmhg"
server_key: "9084490b06074587c3cfff77b6073e30"
server_key_iv: "98ff6f05f56b505ab0b947579205b187"
server_key_version: "encryptionkey@0"
I20260809 04:35:21.833182 25078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:21.854591 25078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:21.857677 25078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:21.857861 25078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:21.858471 25078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:21.859314 25078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:21.859364 25078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:21.859416 25078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:21.859444 25078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:21.868072 25078 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:33785
I20260809 04:35:21.868105 25276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:33785 every 8 connection(s)
I20260809 04:35:21.869051 25078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:21.874969 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 25078
I20260809 04:35:21.875073 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:21.875327 19824 external_mini_cluster.cc:1428] Setting key baae63212c2d6fade9e5d55d9c2d141a
Aug 09 04:35:21 dist-test-slave-rmhg krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250121, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:21.883536 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:21.871052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:44399 (local address 127.19.92.62:36747)
0809 04:35:21.871197 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:21.871201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:21.872079 (+ 878us) server_negotiation.cc:407] Connection header received
0809 04:35:21.873500 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:21.873504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:21.873553 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:21.873628 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:21.874831 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:21.875463 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:21.876791 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:21.877026 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:21.877106 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:21.879837 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:21.879840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:21.879843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:21.879867 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:21.881392 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:21.881872 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:21.881877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:21.881879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:21.882003 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:21.882347 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:21.882350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:21.882351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:21.882594 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:21.882687 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:21.883217 (+ 530us) server_negotiation.cc:299] Negotiation successful
0809 04:35:21.883352 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:35:21.884343 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:21.871441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36747 (local address 127.19.92.1:44399)
0809 04:35:21.871912 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:21.871957 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:21.873268 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:21.873783 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:21.873789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:21.874141 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:21.874662 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:21.874680 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:21.875587 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:21.875591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:21.876671 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:21.876681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:21.877200 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:21.877205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:21.877395 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:21.877974 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:21.877993 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:21.879655 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:21.881503 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:21.881508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:21.881520 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:21.881768 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:21.882103 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:21.882105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:21.882107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:21.882242 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:21.882701 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:21.882709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:21.882981 (+ 272us) client_negotiation.cc:769] Sending connection context
0809 04:35:21.883155 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 04:35:21.883489 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260809 04:35:21.885632 25277 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36747
I20260809 04:35:21.885910 25277 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:35:21.886549 25277 heartbeater.cc:511] Master 127.19.92.62:36747 requested a full tablet report, sending...
I20260809 04:35:21.888193 24994 ts_manager.cc:194] Registered new tserver with Master: 448dbe2c26f545c193068ad28565d57d (127.19.92.1:33785)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:35:21.889714 24994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:44399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:21.904624 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36747
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:22.018478 25283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:22.018740 25283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:22.018780 25283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:22.022773 25283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:22.022823 25283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:22.022881 25283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:22.028088 25283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:22.029333 25283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:22.030419 25283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:22.040036 25289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:22.040009 25296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:22.040112 25283 server_base.cc:1034] running on GCE node
W20260809 04:35:22.040028 25290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:22.040663 25283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:22.041282 25283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:22.042430 25283 hybrid_clock.cc:648] HybridClock initialized: now 1786250122042405 us; error 36 us; skew 500 ppm
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250122, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:22.045024 25283 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:22.045914 25283 webserver.cc:460] Webserver started at http://127.19.92.2:41777/ using document root <none> and password file <none>
I20260809 04:35:22.046442 25283 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:22.046505 25283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:22.046726 25283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:22.048733 25283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "9235bf8c9e7041bc9611f073736a8c2d"
format_stamp: "Formatted at 2026-08-09 04:35:22 on dist-test-slave-rmhg"
server_key: "5399667e5ff41424cd1dc1b6a84c140d"
server_key_iv: "c76177f1141070b2a77709f77b2c6a39"
server_key_version: "encryptionkey@0"
I20260809 04:35:22.049322 25283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "9235bf8c9e7041bc9611f073736a8c2d"
format_stamp: "Formatted at 2026-08-09 04:35:22 on dist-test-slave-rmhg"
server_key: "5399667e5ff41424cd1dc1b6a84c140d"
server_key_iv: "c76177f1141070b2a77709f77b2c6a39"
server_key_version: "encryptionkey@0"
I20260809 04:35:22.052898 25283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:22.055231 25306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:22.056371 25283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:22.056474 25283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "9235bf8c9e7041bc9611f073736a8c2d"
format_stamp: "Formatted at 2026-08-09 04:35:22 on dist-test-slave-rmhg"
server_key: "5399667e5ff41424cd1dc1b6a84c140d"
server_key_iv: "c76177f1141070b2a77709f77b2c6a39"
server_key_version: "encryptionkey@0"
I20260809 04:35:22.056582 25283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:22.066057 25283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:22.069111 25283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:22.069350 25283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:22.069980 25283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:22.070832 25283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:22.070879 25283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:22.070933 25283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:22.070960 25283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:22.079711 25283 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:43833
I20260809 04:35:22.079743 25477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:43833 every 8 connection(s)
I20260809 04:35:22.080610 25283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:22.082347 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 25283
I20260809 04:35:22.082472 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:22.082748 19824 external_mini_cluster.cc:1428] Setting key 79b34c5475de3e0ee737eb9c82663e27
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250122, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:35:22.095350 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:22.082440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:33621 (local address 127.19.92.62:36747)
0809 04:35:22.082581 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:22.082586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:22.083479 (+ 893us) server_negotiation.cc:407] Connection header received
0809 04:35:22.085146 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:22.085178 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:22.085231 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:22.085310 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:22.086814 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.087601 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.088714 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.089051 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.089188 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:22.091853 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:22.091857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:22.091858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:22.091882 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:22.093269 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:22.093773 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:22.093776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:22.093778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:22.093864 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:22.094324 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:22.094327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:22.094328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:22.094527 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:22.094610 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:22.095085 (+ 475us) server_negotiation.cc:299] Negotiation successful
0809 04:35:22.095212 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:35:22.096256 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:22.082830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36747 (local address 127.19.92.2:33621)
0809 04:35:22.083308 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:22.083365 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:22.084920 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:22.085508 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:22.085515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:22.086013 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:22.086670 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:22.086687 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.087702 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.087704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:22.088587 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:22.088596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.089214 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.089218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:22.089396 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:22.089982 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:22.089998 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:22.091664 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:22.093398 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:22.093402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:22.093414 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:22.093660 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:22.093957 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:22.093959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:22.093961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:22.094209 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:22.094640 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:22.094645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:22.094882 (+ 237us) client_negotiation.cc:769] Sending connection context
0809 04:35:22.095050 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 04:35:22.095355 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260809 04:35:22.097445 25478 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36747
I20260809 04:35:22.097674 25478 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:22.098150 25478 heartbeater.cc:511] Master 127.19.92.62:36747 requested a full tablet report, sending...
I20260809 04:35:22.099138 24994 ts_manager.cc:194] Registered new tserver with Master: 9235bf8c9e7041bc9611f073736a8c2d (127.19.92.2:43833)
I20260809 04:35:22.099713 24994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:33621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:22.111896 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36747
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:22.226475 25483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:22.226770 25483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:22.226816 25483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:22.231050 25483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:22.231124 25483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:22.231194 25483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:22.236889 25483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:22.238054 25483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:22.239084 25483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:22.248807 25494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:22.249027 25498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:22.249001 25483 server_base.cc:1034] running on GCE node
W20260809 04:35:22.248800 25492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:22.249538 25483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:22.250146 25483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:22.251300 25483 hybrid_clock.cc:648] HybridClock initialized: now 1786250122251278 us; error 35 us; skew 500 ppm
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250122, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:22.253932 25483 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:22.254773 25483 webserver.cc:460] Webserver started at http://127.19.92.3:40607/ using document root <none> and password file <none>
I20260809 04:35:22.255290 25483 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:22.255348 25483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:22.255513 25483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:22.257577 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "3cf7787ea3aa43c187df2d075edd6148"
format_stamp: "Formatted at 2026-08-09 04:35:22 on dist-test-slave-rmhg"
server_key: "57d392df25fc8ca52320d661d6b77a7a"
server_key_iv: "98401cf8239f0c2adedff4615b9204d9"
server_key_version: "encryptionkey@0"
I20260809 04:35:22.258095 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "3cf7787ea3aa43c187df2d075edd6148"
format_stamp: "Formatted at 2026-08-09 04:35:22 on dist-test-slave-rmhg"
server_key: "57d392df25fc8ca52320d661d6b77a7a"
server_key_iv: "98401cf8239f0c2adedff4615b9204d9"
server_key_version: "encryptionkey@0"
I20260809 04:35:22.261734 25483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:35:22.264132 25508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:22.265573 25483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:22.265683 25483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "3cf7787ea3aa43c187df2d075edd6148"
format_stamp: "Formatted at 2026-08-09 04:35:22 on dist-test-slave-rmhg"
server_key: "57d392df25fc8ca52320d661d6b77a7a"
server_key_iv: "98401cf8239f0c2adedff4615b9204d9"
server_key_version: "encryptionkey@0"
I20260809 04:35:22.265769 25483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:22.290827 25483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:22.294158 25483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:22.294329 25483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:22.294901 25483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:22.295764 25483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:22.295812 25483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:22.295852 25483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:22.295872 25483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:22.304898 25483 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:33653
I20260809 04:35:22.304924 25674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:33653 every 8 connection(s)
I20260809 04:35:22.305927 25483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:22.309514 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 25483
I20260809 04:35:22.309624 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:22.309899 19824 external_mini_cluster.cc:1428] Setting key 7df9b8f50fd6a68f090afc4bfc9d5050
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250122, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:22.319845 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:22.307903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:35197 (local address 127.19.92.62:36747)
0809 04:35:22.308070 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:22.308074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:22.309040 (+ 966us) server_negotiation.cc:407] Connection header received
0809 04:35:22.310727 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:22.310730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:22.310785 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:22.310852 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:22.312411 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.312941 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.314105 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.314321 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.314400 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:22.316735 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:22.316738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:22.316740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:22.316764 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:22.318076 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:22.318551 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:22.318553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:22.318554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:22.318620 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:22.318875 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:22.318877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:22.318878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:22.319057 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:22.319126 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:22.319610 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 04:35:22.319719 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:35:22.320686 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:22.308324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36747 (local address 127.19.92.3:35197)
0809 04:35:22.308857 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:22.308907 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:22.310497 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:22.311043 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:22.311052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:22.311600 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:22.312279 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:22.312295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.313050 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.313052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:22.313998 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:22.314008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.314446 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.314449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:22.314606 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:22.315139 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:22.315155 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:22.316580 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:22.318183 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:22.318188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:22.318202 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:22.318451 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:22.318679 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:22.318681 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:22.318682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:22.318812 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:22.319126 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:22.319131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:22.319371 (+ 240us) client_negotiation.cc:769] Sending connection context
0809 04:35:22.319552 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 04:35:22.319823 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":131,"threads_started":1}
I20260809 04:35:22.321945 25675 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36747
I20260809 04:35:22.322201 25675 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:22.322742 25675 heartbeater.cc:511] Master 127.19.92.62:36747 requested a full tablet report, sending...
I20260809 04:35:22.323868 24992 ts_manager.cc:194] Registered new tserver with Master: 3cf7787ea3aa43c187df2d075edd6148 (127.19.92.3:33653)
I20260809 04:35:22.324467 24992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:35197
I20260809 04:35:22.336018 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:35:22.363726 25686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:22.354487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.19.92.1:33785)
0809 04:35:22.354782 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:22.354800 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:22.354826 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:35:22.354878 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:22.354884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:22.355078 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:22.355199 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:22.355836 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.356624 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.357637 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.358060 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.358158 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:22.360634 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:22.360645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:22.360660 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:22.360686 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:22.362175 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:22.362576 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:22.362581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:22.362583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:22.362663 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:22.362849 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:22.362851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:22.362852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:22.363096 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:22.363269 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:22.363402 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 04:35:22.363566 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[24927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:22.375015 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:22.366945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.19.92.62:36747)
0809 04:35:22.367096 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:22.367099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:22.367113 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:22.367195 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:22.367198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:22.367239 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:22.367300 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:22.367915 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.368433 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.369245 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:22.369489 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:22.369577 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:22.372072 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:22.372075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:22.372077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:22.372105 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:22.373457 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:22.373813 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:22.373816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:22.373817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:22.373889 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:22.374226 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:22.374228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:22.374229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:22.374460 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:22.374567 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:22.374747 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 04:35:22.374849 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 04:35:22.375581 24992 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44668
I20260809 04:35:22.376300 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 25078
I20260809 04:35:22.389842 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 25283
I20260809 04:35:22.400645 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 25483
I20260809 04:35:22.416571 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 24946
2026-08-09T04:35:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2291 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[25694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[25694](info): set up 2 sockets
Aug 09 04:35:22 dist-test-slave-rmhg krb5kdc[25694](info): commencing operation
krb5kdc: starting...
W20260809 04:35:23.540145 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:23 dist-test-slave-rmhg krb5kdc[25694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:23.663516 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:35589
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:35589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:23.775400 25712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:23.775668 25712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:23.775714 25712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:23.779711 25712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:23.779769 25712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:23.779788 25712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:23.779801 25712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:23.779819 25712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:23.785177 25712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:35589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:35589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:23.786370 25712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:23.787514 25712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:23.796222 25725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:23.796241 25728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:23.796223 25721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:23.796576 25712 server_base.cc:1034] running on GCE node
I20260809 04:35:23.797190 25712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:23.798389 25712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:23.799551 25712 hybrid_clock.cc:648] HybridClock initialized: now 1786250123799531 us; error 30 us; skew 500 ppm
Aug 09 04:35:23 dist-test-slave-rmhg krb5kdc[25694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:23.802268 25712 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:23.802994 25712 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:23.803058 25712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:23.803236 25712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:23.805334 25712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "20913d6231bc47edb201f9a7aed3c9a1"
format_stamp: "Formatted at 2026-08-09 04:35:23 on dist-test-slave-rmhg"
server_key: "380a762ebb0e51e67d026350bff148cc"
server_key_iv: "0cc92e67a2064d01adafd0f55f6e6069"
server_key_version: "encryptionkey@0"
I20260809 04:35:23.805852 25712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "20913d6231bc47edb201f9a7aed3c9a1"
format_stamp: "Formatted at 2026-08-09 04:35:23 on dist-test-slave-rmhg"
server_key: "380a762ebb0e51e67d026350bff148cc"
server_key_iv: "0cc92e67a2064d01adafd0f55f6e6069"
server_key_version: "encryptionkey@0"
I20260809 04:35:23.809599 25712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:35:23.812186 25737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:23.813697 25712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:23.813833 25712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "20913d6231bc47edb201f9a7aed3c9a1"
format_stamp: "Formatted at 2026-08-09 04:35:23 on dist-test-slave-rmhg"
server_key: "380a762ebb0e51e67d026350bff148cc"
server_key_iv: "0cc92e67a2064d01adafd0f55f6e6069"
server_key_version: "encryptionkey@0"
I20260809 04:35:23.813937 25712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:23.831059 25712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:23.834131 25712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:23.834316 25712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:23.841064 25712 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:35589
I20260809 04:35:23.841323 25808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:35589 every 8 connection(s)
I20260809 04:35:23.841753 25712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:23.844471 25810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:23.850332 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 25712
I20260809 04:35:23.850443 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:23.850519 25810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1: Bootstrap starting.
I20260809 04:35:23.850708 19824 external_mini_cluster.cc:1428] Setting key 12205c0491247bcc5728497a95db62e6
I20260809 04:35:23.853243 25810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:23.854175 25810 log.cc:824] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:23.856371 25810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1: No bootstrap required, opened a new log
Aug 09 04:35:23 dist-test-slave-rmhg krb5kdc[25694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:23.860253 25810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20913d6231bc47edb201f9a7aed3c9a1" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35589 } }
I20260809 04:35:23.860476 25810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:23.860553 25810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20913d6231bc47edb201f9a7aed3c9a1, State: Initialized, Role: FOLLOWER
I20260809 04:35:23.860996 25810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20913d6231bc47edb201f9a7aed3c9a1" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35589 } }
I20260809 04:35:23.861193 25810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:23.861272 25810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:23.861379 25810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:23.862277 25810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20913d6231bc47edb201f9a7aed3c9a1" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35589 } }
I20260809 04:35:23.862607 25810 leader_election.cc:302] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20913d6231bc47edb201f9a7aed3c9a1; no voters:
I20260809 04:35:23.862916 25810 leader_election.cc:288] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:23.863013 25818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:23.863214 25818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [term 1 LEADER]: Becoming Leader. State: Replica: 20913d6231bc47edb201f9a7aed3c9a1, State: Running, Role: LEADER
I20260809 04:35:23.863569 25818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20913d6231bc47edb201f9a7aed3c9a1" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35589 } }
I20260809 04:35:23.864228 25810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:23.865402 25819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20913d6231bc47edb201f9a7aed3c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20913d6231bc47edb201f9a7aed3c9a1" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35589 } } }
I20260809 04:35:23.865530 25819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:23.865587 25818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20913d6231bc47edb201f9a7aed3c9a1. Latest consensus state: current_term: 1 leader_uuid: "20913d6231bc47edb201f9a7aed3c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20913d6231bc47edb201f9a7aed3c9a1" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35589 } } }
I20260809 04:35:23.865712 25818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:23.865514 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:23.852106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.19.92.62:35589)
0809 04:35:23.852533 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:23.852553 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:23.852595 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:35:23.853457 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:23.853485 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:23.853908 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:23.854282 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:23.855084 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:23.856497 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:23.857353 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:23.858087 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:23.858246 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:23.860599 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:23.860606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:23.860624 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:23.860658 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:23.862426 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:23.862845 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:23.862852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:23.862858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:23.862978 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:23.863293 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:23.863296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:23.863298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:23.863758 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:23.863962 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:23.864258 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 04:35:23.864582 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":111,"threads_started":1}
I20260809 04:35:23.866935 25832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:23.870267 25832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:23.875663 25832 catalog_manager.cc:1347] Generated new cluster ID: 38d75400f60a4595b7ae7517b832fc83
I20260809 04:35:23.875739 25832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:23.886560 25832 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:23.887897 25832 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:23.906420 25832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1: Generated new TSK 0
I20260809 04:35:23.907389 25832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:23.941519 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35589
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:24.053855 25850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:24.054102 25850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:24.054145 25850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:24.058063 25850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:24.058112 25850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:24.058171 25850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:24.063212 25850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:24.064258 25850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:24.065379 25850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:24.075070 25863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:24.075070 25858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:24.075070 25859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:24.075430 25850 server_base.cc:1034] running on GCE node
I20260809 04:35:24.075889 25850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:24.076562 25850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:24.077729 25850 hybrid_clock.cc:648] HybridClock initialized: now 1786250124077712 us; error 36 us; skew 500 ppm
Aug 09 04:35:24 dist-test-slave-rmhg krb5kdc[25694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250124, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:24.080458 25850 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:24.081233 25850 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:24.081311 25850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:24.081523 25850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:24.083563 25850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "1b6d1870fb474e47b6c9b8a64bdfe641"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "98e831a6bcc1d6a149061a117e7116eb"
server_key_iv: "9e3a918a63497cf9761210de1d556b7e"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.084098 25850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "1b6d1870fb474e47b6c9b8a64bdfe641"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "98e831a6bcc1d6a149061a117e7116eb"
server_key_iv: "9e3a918a63497cf9761210de1d556b7e"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.087725 25850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 04:35:24.090169 25870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.091383 25850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:24.091502 25850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "1b6d1870fb474e47b6c9b8a64bdfe641"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "98e831a6bcc1d6a149061a117e7116eb"
server_key_iv: "9e3a918a63497cf9761210de1d556b7e"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.091615 25850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:24.108206 25850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:24.111552 25850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:24.111755 25850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:24.112380 25850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:24.113312 25850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:24.113361 25850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.113417 25850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:24.113442 25850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.122088 25850 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:40791
I20260809 04:35:24.122107 26031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:40791 every 8 connection(s)
I20260809 04:35:24.122747 25850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:24.128989 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 25850
I20260809 04:35:24.129109 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:24.129391 19824 external_mini_cluster.cc:1428] Setting key b2c21b8c96ebfc8b632c303b545b3cc1
Aug 09 04:35:24 dist-test-slave-rmhg krb5kdc[25694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250124, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:24.138245 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.124471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:43891 (local address 127.19.92.62:35589)
0809 04:35:24.124617 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.124622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.125841 (+ 1219us) server_negotiation.cc:407] Connection header received
0809 04:35:24.126688 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.126691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.126738 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.126798 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:24.127989 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.128541 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.129785 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.130023 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.130118 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.134507 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:24.134510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:24.134513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:24.134544 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:24.136123 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:24.136630 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:24.136634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:24.136636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:24.136735 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:24.137059 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:24.137063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:24.137065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:24.137337 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:24.137441 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.137926 (+ 485us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.138071 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:35:24.138983 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.124902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35589 (local address 127.19.92.1:43891)
0809 04:35:24.125699 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.125745 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:24.126487 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:24.126953 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:24.126960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:24.127308 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:24.127814 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.127827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.128663 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.128666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.129668 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.129678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.130166 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.130169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.130327 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.132781 (+ 2454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:24.132821 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:24.134331 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:24.136235 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:24.136240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:24.136253 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:24.136518 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:24.136817 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:24.136819 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:24.136821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:24.136964 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:24.137432 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:24.137439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:24.137686 (+ 247us) client_negotiation.cc:769] Sending connection context
0809 04:35:24.137836 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 04:35:24.138108 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:35:24.140241 26032 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35589
I20260809 04:35:24.140480 26032 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:24.140959 26032 heartbeater.cc:511] Master 127.19.92.62:35589 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:35:24.142594 25756 ts_manager.cc:194] Registered new tserver with Master: 1b6d1870fb474e47b6c9b8a64bdfe641 (127.19.92.1:40791)
I20260809 04:35:24.143741 25756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:43891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:24.157915 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35589
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:24.269824 26041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:24.270120 26041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:24.270167 26041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:24.274510 26041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:24.274572 26041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:24.274647 26041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:24.280025 26041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:24.281121 26041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:24.282184 26041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:35:24.291714 26041 server_base.cc:1034] running on GCE node
W20260809 04:35:24.291639 26052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:24.291658 26047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:24.291658 26048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:24.292340 26041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:24.292999 26041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:24.294178 26041 hybrid_clock.cc:648] HybridClock initialized: now 1786250124294145 us; error 46 us; skew 500 ppm
Aug 09 04:35:24 dist-test-slave-rmhg krb5kdc[25694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250124, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:24.300171 26041 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:24.300902 26041 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:24.300974 26041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:24.301219 26041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:24.303277 26041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "4c009a4e198a42ba8596062bb13462e5"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "077dcf849c70ff26fc88e3ac25da28eb"
server_key_iv: "3072d28371a78645f92cbb7eae1aec3d"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.303828 26041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "4c009a4e198a42ba8596062bb13462e5"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "077dcf849c70ff26fc88e3ac25da28eb"
server_key_iv: "3072d28371a78645f92cbb7eae1aec3d"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.307644 26041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 04:35:24.310433 26059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.311798 26041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:24.311923 26041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "4c009a4e198a42ba8596062bb13462e5"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "077dcf849c70ff26fc88e3ac25da28eb"
server_key_iv: "3072d28371a78645f92cbb7eae1aec3d"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.312052 26041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:24.327538 26041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:24.330399 26041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:24.330569 26041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:24.331164 26041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:24.332165 26041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:24.332226 26041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.332279 26041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:24.332309 26041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.341061 26041 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:43869
I20260809 04:35:24.341094 26226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:43869 every 8 connection(s)
I20260809 04:35:24.341655 26041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:24.345340 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 26041
I20260809 04:35:24.345455 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:24.345798 19824 external_mini_cluster.cc:1428] Setting key 2d57e5aeb65ad50cd6a2c9860ff002c1
Aug 09 04:35:24 dist-test-slave-rmhg krb5kdc[25694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250124, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:24.356329 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.343333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:58127 (local address 127.19.92.62:35589)
0809 04:35:24.343480 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.343484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.344620 (+ 1136us) server_negotiation.cc:407] Connection header received
0809 04:35:24.345536 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.345539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.345597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.345692 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:24.346931 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.347534 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.349063 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.349391 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.349492 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.352164 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:24.352169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:24.352171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:24.352205 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:24.353988 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:24.354584 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:24.354587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:24.354589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:24.354692 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:24.355029 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:24.355033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:24.355035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:24.355295 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:24.355399 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.356001 (+ 602us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.356151 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:35:24.357227 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.343685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35589 (local address 127.19.92.2:58127)
0809 04:35:24.344478 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.344521 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:24.345329 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:24.345840 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:24.345849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:24.346224 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:24.346796 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.346810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.347669 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.347672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.348956 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.348969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.349537 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.349540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.349699 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.350278 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:24.350295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:24.351967 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:24.354144 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:24.354150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:24.354165 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:24.354453 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:24.354781 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:24.354783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:24.354784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:24.354936 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:24.355420 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:24.355427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:24.355729 (+ 302us) client_negotiation.cc:769] Sending connection context
0809 04:35:24.355905 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 04:35:24.356210 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260809 04:35:24.358604 26228 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35589
I20260809 04:35:24.358946 26228 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:35:24.359543 26228 heartbeater.cc:511] Master 127.19.92.62:35589 requested a full tablet report, sending...
I20260809 04:35:24.360823 25756 ts_manager.cc:194] Registered new tserver with Master: 4c009a4e198a42ba8596062bb13462e5 (127.19.92.2:43869)
I20260809 04:35:24.361508 25756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:58127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:24.375435 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35589
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:24.490747 26242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:24.490990 26242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:24.491034 26242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:24.495043 26242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:24.495103 26242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:24.495162 26242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:24.500352 26242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:24.501355 26242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:24.502305 26242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:24.511986 26255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:24.511988 26251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:24.511986 26252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:24.512082 26242 server_base.cc:1034] running on GCE node
I20260809 04:35:24.512722 26242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:24.513451 26242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:24.514621 26242 hybrid_clock.cc:648] HybridClock initialized: now 1786250124514598 us; error 33 us; skew 500 ppm
Aug 09 04:35:24 dist-test-slave-rmhg krb5kdc[25694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250124, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:24.517534 26242 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:24.518263 26242 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:24.518332 26242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:24.518558 26242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:24.520599 26242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "e6fe202c0d904881b757bb3a52af8ef3"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "d23ee4c2fc2f336357ad0f71beec005a"
server_key_iv: "f6a588001f3703905a460b7c87be1ce7"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.521145 26242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "e6fe202c0d904881b757bb3a52af8ef3"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "d23ee4c2fc2f336357ad0f71beec005a"
server_key_iv: "f6a588001f3703905a460b7c87be1ce7"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.524731 26242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:24.527042 26266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.528322 26242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:24.528469 26242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "e6fe202c0d904881b757bb3a52af8ef3"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "d23ee4c2fc2f336357ad0f71beec005a"
server_key_iv: "f6a588001f3703905a460b7c87be1ce7"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.528565 26242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:24.544184 26242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:24.547389 26242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:24.547590 26242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:24.548211 26242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:24.549100 26242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:24.549178 26242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.549252 26242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:24.549283 26242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:24.558003 26242 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:43471
I20260809 04:35:24.558048 26430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:43471 every 8 connection(s)
I20260809 04:35:24.558647 26242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:24.563877 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 26242
I20260809 04:35:24.564025 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:24.564339 19824 external_mini_cluster.cc:1428] Setting key f814cee8d60519497d87255b94c62a70
Aug 09 04:35:24 dist-test-slave-rmhg krb5kdc[25694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250124, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:24.572036 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.560374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:38261 (local address 127.19.92.62:35589)
0809 04:35:24.560518 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.560522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.561759 (+ 1237us) server_negotiation.cc:407] Connection header received
0809 04:35:24.562641 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.562645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.562706 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.562780 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:24.564172 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.564802 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.566203 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.566429 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.566517 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.568810 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:24.568814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:24.568816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:24.568842 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:24.570053 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:24.570525 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:24.570530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:24.570531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:24.570610 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:24.570924 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:24.570928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:24.570930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:24.571123 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:24.571213 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.571723 (+ 510us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.571858 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:35:24.572835 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.560762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35589 (local address 127.19.92.3:38261)
0809 04:35:24.561612 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.561654 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:24.562430 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:24.562961 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:24.562968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:24.563325 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:24.563974 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.563996 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.564932 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.564935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.566063 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.566074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.566577 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.566581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.566746 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.567237 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:24.567253 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:24.568651 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:24.570159 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:24.570165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:24.570181 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:24.570430 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:24.570692 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:24.570695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:24.570697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:24.570836 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:24.571226 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:24.571234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:24.571484 (+ 250us) client_negotiation.cc:769] Sending connection context
0809 04:35:24.571661 (+ 177us) client_negotiation.cc:240] Negotiation successful
0809 04:35:24.572002 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260809 04:35:24.573951 26431 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35589
I20260809 04:35:24.574173 26431 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:24.574611 26431 heartbeater.cc:511] Master 127.19.92.62:35589 requested a full tablet report, sending...
I20260809 04:35:24.575579 25756 ts_manager.cc:194] Registered new tserver with Master: e6fe202c0d904881b757bb3a52af8ef3 (127.19.92.3:43471)
I20260809 04:35:24.576129 25756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:38261
I20260809 04:35:24.578362 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:35:24.588140 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.580310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.19.92.62:35589)
0809 04:35:24.580445 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.580450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.580621 (+ 171us) server_negotiation.cc:407] Connection header received
0809 04:35:24.580764 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.580767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.580813 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.580873 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:24.581642 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.582165 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.583037 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.583267 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.583355 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.585207 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:24.585211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:24.585213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:24.585242 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:24.586574 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:24.586978 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:24.586981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:24.586982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:24.587061 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:24.587294 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:24.587298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:24.587299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:24.587553 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:24.587663 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.587847 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.587972 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:35:24.591279 25756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:24.594589 25756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:24.609125 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.602994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43471 (local address 127.0.0.1:54212)
0809 04:35:24.603365 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.603421 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:24.603580 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:24.604652 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:24.604658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:24.604696 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:24.604879 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.604887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.606448 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.606452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.607636 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.607646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.608625 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.608628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.608717 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.608749 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:35:24.608836 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 04:35:24.608960 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":109,"spinlock_wait_cycles":6784,"thread_start_us":89,"threads_started":1}
I20260809 04:35:24.609292 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.603326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:43869 (local address 127.0.0.1:34816)
0809 04:35:24.603821 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.603833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:24.603909 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:24.604651 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:24.604658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:24.604697 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:24.604876 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.604883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.606286 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.606288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.607893 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.607904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.608923 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.608926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.609014 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.609037 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:35:24.609101 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 04:35:24.609182 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":326,"spinlock_wait_cycles":6912,"thread_start_us":113,"threads_started":1}
I20260809 04:35:24.609210 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.603122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.19.92.3:43471)
0809 04:35:24.604133 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.604144 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.604167 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:35:24.604242 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.604247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.604467 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.605055 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:24.605255 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.606332 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.607752 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.608515 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.608629 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.608785 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.608895 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.609003 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":152,"threads_started":1}
I20260809 04:35:24.609776 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.603687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.19.92.2:43869)
0809 04:35:24.604132 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.604144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.604167 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:35:24.604242 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.604247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.604466 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.604774 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:24.604970 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.606162 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.608023 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.608802 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.608906 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.609039 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.609537 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.609631 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":88,"threads_started":1}
I20260809 04:35:24.610610 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.603742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:40791 (local address 127.0.0.1:56088)
0809 04:35:24.604550 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.604562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:24.604651 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:24.605196 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:24.605199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:24.605218 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:24.605309 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.605315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.606237 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.606240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.607893 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.607904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.610196 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.610200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.610309 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.610333 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:35:24.610401 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 04:35:24.610472 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":82,"thread_start_us":90,"threads_started":1}
I20260809 04:35:24.611009 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.603904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.19.92.1:40791)
0809 04:35:24.604289 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.604296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.604746 (+ 450us) server_negotiation.cc:407] Connection header received
0809 04:35:24.604826 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.604831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.605051 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.605210 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:24.605410 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.606119 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.609104 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.609911 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.610006 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.610132 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.610772 (+ 640us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.610871 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":94,"threads_started":1}
I20260809 04:35:24.612600 26125 tablet_service.cc:1467] Processing CreateTablet for tablet 30759bc9a43c4465a04358a991bbb21d (DEFAULT_TABLE table=test-table [id=ba537ce6499f4ed39d5004aae93efae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:24.612597 26335 tablet_service.cc:1467] Processing CreateTablet for tablet 30759bc9a43c4465a04358a991bbb21d (DEFAULT_TABLE table=test-table [id=ba537ce6499f4ed39d5004aae93efae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:24.613814 26125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30759bc9a43c4465a04358a991bbb21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:24.613821 26335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30759bc9a43c4465a04358a991bbb21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:24.615768 25938 tablet_service.cc:1467] Processing CreateTablet for tablet 30759bc9a43c4465a04358a991bbb21d (DEFAULT_TABLE table=test-table [id=ba537ce6499f4ed39d5004aae93efae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:24.616719 25938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30759bc9a43c4465a04358a991bbb21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:24.622227 26455 tablet_bootstrap.cc:492] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: Bootstrap starting.
I20260809 04:35:24.623093 26457 tablet_bootstrap.cc:492] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641: Bootstrap starting.
I20260809 04:35:24.621579 26456 tablet_bootstrap.cc:492] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: Bootstrap starting.
I20260809 04:35:24.625028 26457 tablet_bootstrap.cc:654] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:24.625438 26456 tablet_bootstrap.cc:654] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:24.626084 26457 log.cc:824] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:24.626274 26456 log.cc:824] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:24.626600 26455 tablet_bootstrap.cc:654] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:24.627643 26455 log.cc:824] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:24.628228 26457 tablet_bootstrap.cc:492] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641: No bootstrap required, opened a new log
I20260809 04:35:24.628278 26456 tablet_bootstrap.cc:492] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: No bootstrap required, opened a new log
I20260809 04:35:24.628443 26457 ts_tablet_manager.cc:1397] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:35:24.628443 26456 ts_tablet_manager.cc:1397] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260809 04:35:24.629669 26455 tablet_bootstrap.cc:492] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: No bootstrap required, opened a new log
I20260809 04:35:24.629869 26455 ts_tablet_manager.cc:1397] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260809 04:35:24.632593 26456 raft_consensus.cc:348] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.632593 26457 raft_consensus.cc:348] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.632889 26456 raft_consensus.cc:374] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:24.632889 26457 raft_consensus.cc:374] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:24.632962 26456 raft_consensus.cc:729] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c009a4e198a42ba8596062bb13462e5, State: Initialized, Role: FOLLOWER
I20260809 04:35:24.632957 26457 raft_consensus.cc:729] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6d1870fb474e47b6c9b8a64bdfe641, State: Initialized, Role: FOLLOWER
I20260809 04:35:24.632936 26455 raft_consensus.cc:348] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.633307 26455 raft_consensus.cc:374] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:24.633368 26455 raft_consensus.cc:729] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fe202c0d904881b757bb3a52af8ef3, State: Initialized, Role: FOLLOWER
I20260809 04:35:24.633483 26457 consensus_queue.cc:260] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.633483 26456 consensus_queue.cc:260] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.633775 26455 consensus_queue.cc:260] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.634593 26032 heartbeater.cc:503] Master 127.19.92.62:35589 was elected leader, sending a full tablet report...
I20260809 04:35:24.634598 26228 heartbeater.cc:503] Master 127.19.92.62:35589 was elected leader, sending a full tablet report...
I20260809 04:35:24.634617 26457 ts_tablet_manager.cc:1428] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 04:35:24.634610 26456 ts_tablet_manager.cc:1428] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:35:24.636005 26455 ts_tablet_manager.cc:1428] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:35:24.636348 26431 heartbeater.cc:503] Master 127.19.92.62:35589 was elected leader, sending a full tablet report...
I20260809 04:35:24.691813 26463 raft_consensus.cc:482] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:24.692039 26463 raft_consensus.cc:504] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.693220 26463 leader_election.cc:288] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6fe202c0d904881b757bb3a52af8ef3 (127.19.92.3:43471), 1b6d1870fb474e47b6c9b8a64bdfe641 (127.19.92.1:40791)
I20260809 04:35:24.696870 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.693502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43471 (local address 127.19.92.2:58761)
0809 04:35:24.693642 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.693658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:24.693746 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:24.694039 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:24.694042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:24.694065 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:24.694171 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.694177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.694921 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.694924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.695826 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.695836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.696465 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.696469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.696581 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.696602 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:35:24.696666 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 04:35:24.696737 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 04:35:24.696870 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.693600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:58761 (local address 127.19.92.3:43471)
0809 04:35:24.693724 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.693728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.693746 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:35:24.693860 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.693863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.693920 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.694000 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:24.694285 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.694825 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.695949 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.696361 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.696453 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.696529 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.696694 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.696759 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:24.697343 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.693770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:40791 (local address 127.19.92.2:36391)
0809 04:35:24.694153 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.694164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:24.694256 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:24.694489 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:24.694491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:24.694509 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:24.694628 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.694633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.695386 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.695389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.696274 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:24.696284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.696907 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.696911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:24.697013 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.697036 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:35:24.697128 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 04:35:24.697209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":135,"threads_started":1}
I20260809 04:35:24.697376 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.693912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:36391 (local address 127.19.92.1:40791)
0809 04:35:24.694056 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.694060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.694231 (+ 171us) server_negotiation.cc:407] Connection header received
0809 04:35:24.694341 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.694344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.694387 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.694457 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:24.694748 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.695289 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.696385 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.696785 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.696876 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.696936 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.697182 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.697262 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:35:24.697705 26369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30759bc9a43c4465a04358a991bbb21d" candidate_uuid: "4c009a4e198a42ba8596062bb13462e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fe202c0d904881b757bb3a52af8ef3" is_pre_election: true
I20260809 04:35:24.698001 25970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30759bc9a43c4465a04358a991bbb21d" candidate_uuid: "4c009a4e198a42ba8596062bb13462e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" is_pre_election: true
I20260809 04:35:24.698038 26369 raft_consensus.cc:2454] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c009a4e198a42ba8596062bb13462e5 in term 0.
I20260809 04:35:24.698248 25970 raft_consensus.cc:2454] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c009a4e198a42ba8596062bb13462e5 in term 0.
I20260809 04:35:24.698518 26065 leader_election.cc:302] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c009a4e198a42ba8596062bb13462e5, e6fe202c0d904881b757bb3a52af8ef3; no voters:
I20260809 04:35:24.698804 26463 raft_consensus.cc:2781] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:24.698881 26463 raft_consensus.cc:482] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:24.698910 26463 raft_consensus.cc:3037] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:24.699903 26463 raft_consensus.cc:504] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.700207 26463 leader_election.cc:288] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [CANDIDATE]: Term 1 election: Requested vote from peers e6fe202c0d904881b757bb3a52af8ef3 (127.19.92.3:43471), 1b6d1870fb474e47b6c9b8a64bdfe641 (127.19.92.1:40791)
I20260809 04:35:24.700539 26369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30759bc9a43c4465a04358a991bbb21d" candidate_uuid: "4c009a4e198a42ba8596062bb13462e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fe202c0d904881b757bb3a52af8ef3"
I20260809 04:35:24.700608 25970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30759bc9a43c4465a04358a991bbb21d" candidate_uuid: "4c009a4e198a42ba8596062bb13462e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641"
I20260809 04:35:24.700650 26369 raft_consensus.cc:3037] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:24.700721 25970 raft_consensus.cc:3037] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:24.701745 25970 raft_consensus.cc:2454] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c009a4e198a42ba8596062bb13462e5 in term 1.
I20260809 04:35:24.701753 26369 raft_consensus.cc:2454] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c009a4e198a42ba8596062bb13462e5 in term 1.
I20260809 04:35:24.702030 26065 leader_election.cc:302] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c009a4e198a42ba8596062bb13462e5, e6fe202c0d904881b757bb3a52af8ef3; no voters:
I20260809 04:35:24.702193 26463 raft_consensus.cc:2781] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:24.702418 26463 raft_consensus.cc:686] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 1 LEADER]: Becoming Leader. State: Replica: 4c009a4e198a42ba8596062bb13462e5, State: Running, Role: LEADER
I20260809 04:35:24.702769 26463 consensus_queue.cc:237] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } }
I20260809 04:35:24.705876 25755 catalog_manager.cc:5495] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c009a4e198a42ba8596062bb13462e5 (127.19.92.2). New cstate: current_term: 1 leader_uuid: "4c009a4e198a42ba8596062bb13462e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c009a4e198a42ba8596062bb13462e5" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 43869 } health_report { overall_health: HEALTHY } } }
I20260809 04:35:24.734839 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.731431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.19.92.2:43869)
0809 04:35:24.731572 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:24.731575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:24.731590 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:35:24.731665 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:24.731667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:24.731723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:24.731812 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:24.731990 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.732548 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.733380 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:24.733626 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:24.733702 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:24.733987 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:24.734483 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:24.734577 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:24.734655 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 04:35:24.734706 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:35:24.740489 26369 raft_consensus.cc:1261] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 1 FOLLOWER]: Refusing update from remote peer 4c009a4e198a42ba8596062bb13462e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:24.740485 25970 raft_consensus.cc:1261] T 30759bc9a43c4465a04358a991bbb21d P 1b6d1870fb474e47b6c9b8a64bdfe641 [term 1 FOLLOWER]: Refusing update from remote peer 4c009a4e198a42ba8596062bb13462e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:24.741174 26471 consensus_queue.cc:1035] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6fe202c0d904881b757bb3a52af8ef3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:24.741359 26463 consensus_queue.cc:1035] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b6d1870fb474e47b6c9b8a64bdfe641" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:24.748349 26474 mvcc.cc:205] Tried to move back new op lower bound from 7316480510930161664 to 7316480510785314816. Current Snapshot: MvccSnapshot[applied={T|T < 7316480510930161664}]
I20260809 04:35:24.758062 25755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:24.758230 25755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:24.760691 25755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20913d6231bc47edb201f9a7aed3c9a1: Sending DeleteTablet for 3 replicas of tablet 30759bc9a43c4465a04358a991bbb21d
I20260809 04:35:24.761478 26335 tablet_service.cc:1514] Processing DeleteTablet for tablet 30759bc9a43c4465a04358a991bbb21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:24 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:54212
I20260809 04:35:24.761549 26125 tablet_service.cc:1514] Processing DeleteTablet for tablet 30759bc9a43c4465a04358a991bbb21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:24 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:34816
I20260809 04:35:24.761996 26487 tablet_replica.cc:330] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: stopping tablet replica
I20260809 04:35:24.762243 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 25850
I20260809 04:35:24.762320 25938 tablet_service.cc:1514] Processing DeleteTablet for tablet 30759bc9a43c4465a04358a991bbb21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:24 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:56088
I20260809 04:35:24.766534 26487 raft_consensus.cc:2229] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:24.766916 26487 raft_consensus.cc:2258] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:24.771214 26488 tablet_replica.cc:330] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: stopping tablet replica
I20260809 04:35:24.771559 26488 raft_consensus.cc:2229] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:24.771857 26487 ts_tablet_manager.cc:1905] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:24.772007 26488 raft_consensus.cc:2258] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:24.773691 26488 ts_tablet_manager.cc:1905] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:24.774717 26487 ts_tablet_manager.cc:1918] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:24.774830 26487 log.cc:1196] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/30759bc9a43c4465a04358a991bbb21d
I20260809 04:35:24.775171 26487 ts_tablet_manager.cc:1939] T 30759bc9a43c4465a04358a991bbb21d P e6fe202c0d904881b757bb3a52af8ef3: Deleting consensus metadata
I20260809 04:35:24.776505 26488 ts_tablet_manager.cc:1918] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:24.776551 25741 catalog_manager.cc:4841] TS e6fe202c0d904881b757bb3a52af8ef3 (127.19.92.3:43471): tablet 30759bc9a43c4465a04358a991bbb21d (table test-table [id=ba537ce6499f4ed39d5004aae93efae4]) successfully deleted
I20260809 04:35:24.776628 26488 log.cc:1196] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/30759bc9a43c4465a04358a991bbb21d
I20260809 04:35:24.777004 26488 ts_tablet_manager.cc:1939] T 30759bc9a43c4465a04358a991bbb21d P 4c009a4e198a42ba8596062bb13462e5: Deleting consensus metadata
W20260809 04:35:24.777851 25741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:40791 (error 108)
I20260809 04:35:24.778200 25742 catalog_manager.cc:4841] TS 4c009a4e198a42ba8596062bb13462e5 (127.19.92.2:43869): tablet 30759bc9a43c4465a04358a991bbb21d (table test-table [id=ba537ce6499f4ed39d5004aae93efae4]) successfully deleted
I20260809 04:35:24.778749 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 26041
I20260809 04:35:24.778944 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:24.778640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:40791 (local address 127.0.0.1:56096)
0809 04:35:24.778770 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:24.778847 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:40791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 04:35:24.788888 25741 catalog_manager.cc:4584] TS 1b6d1870fb474e47b6c9b8a64bdfe641 (127.19.92.1:40791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30759bc9a43c4465a04358a991bbb21d: Network error: Client connection negotiation failed: client connection to 127.19.92.1:40791: connect: Connection refused (error 111)
I20260809 04:35:24.799443 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 26242
I20260809 04:35:24.817881 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 25712
2026-08-09T04:35:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2407 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T04:35:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:24Z Disabled control of system clock
I20260809 04:35:24.856562 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36679
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 04:35:24.968525 26495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:24.968788 26495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:24.968825 26495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:24.972702 26495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:24.972751 26495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:24.972764 26495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:24.972779 26495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:24.972792 26495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:24.977406 26495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:24.978394 26495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:24.979718 26495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:24.987984 26507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:24.987988 26503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:24.987984 26501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:24.988054 26495 server_base.cc:1034] running on GCE node
I20260809 04:35:24.988873 26495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:24.990202 26495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:24.991384 26495 hybrid_clock.cc:648] HybridClock initialized: now 1786250124991368 us; error 35 us; skew 500 ppm
I20260809 04:35:24.993479 26495 webserver.cc:460] Webserver started at http://127.19.92.62:39271/ using document root <none> and password file <none>
I20260809 04:35:24.994082 26495 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:24.994155 26495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:24.994379 26495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:24.996526 26495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "58aa6bf3347e4c67bd6ab657353a30f2"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "80d404848796b0f328220a11ddb7e639"
server_key_iv: "ca0207aebe0a0db221348b9ae09181e4"
server_key_version: "encryptionkey@0"
I20260809 04:35:24.997084 26495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "58aa6bf3347e4c67bd6ab657353a30f2"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "80d404848796b0f328220a11ddb7e639"
server_key_iv: "ca0207aebe0a0db221348b9ae09181e4"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.000837 26495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:25.003717 26524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.004977 26495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:25.005095 26495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "58aa6bf3347e4c67bd6ab657353a30f2"
format_stamp: "Formatted at 2026-08-09 04:35:24 on dist-test-slave-rmhg"
server_key: "80d404848796b0f328220a11ddb7e639"
server_key_iv: "ca0207aebe0a0db221348b9ae09181e4"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.005227 26495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:25.009812 26495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:25.010404 26495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:25.010569 26495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:25.016943 26495 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:36679
I20260809 04:35:25.016963 26606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:36679 every 8 connection(s)
I20260809 04:35:25.017867 26495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:25.021006 26607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:25.022184 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 26495
I20260809 04:35:25.022315 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:25.022644 19824 external_mini_cluster.cc:1428] Setting key aafe2eaeadbc9ad90208203bf79dcc13
I20260809 04:35:25.027424 26607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2: Bootstrap starting.
I20260809 04:35:25.028541 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.024163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.19.92.62:36679)
0809 04:35:25.024854 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.024869 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.024905 (+ 36us) server_negotiation.cc:407] Connection header received
0809 04:35:25.025598 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.025616 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.025973 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.026291 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.027081 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.027085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.027097 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.027108 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.027243 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.027262 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.027402 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.027475 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.027834 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":167,"threads_started":1}
I20260809 04:35:25.029807 26607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:25.030614 26607 log.cc:824] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:25.032790 26607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2: No bootstrap required, opened a new log
I20260809 04:35:25.036530 26607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58aa6bf3347e4c67bd6ab657353a30f2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36679 } }
I20260809 04:35:25.036758 26607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:25.036799 26607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aa6bf3347e4c67bd6ab657353a30f2, State: Initialized, Role: FOLLOWER
I20260809 04:35:25.037223 26607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58aa6bf3347e4c67bd6ab657353a30f2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36679 } }
I20260809 04:35:25.037357 26607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:25.037406 26607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:25.037485 26607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:25.038440 26607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58aa6bf3347e4c67bd6ab657353a30f2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36679 } }
I20260809 04:35:25.038763 26607 leader_election.cc:302] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58aa6bf3347e4c67bd6ab657353a30f2; no voters:
I20260809 04:35:25.039060 26607 leader_election.cc:288] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:25.039156 26612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:25.039371 26612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [term 1 LEADER]: Becoming Leader. State: Replica: 58aa6bf3347e4c67bd6ab657353a30f2, State: Running, Role: LEADER
I20260809 04:35:25.039712 26612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58aa6bf3347e4c67bd6ab657353a30f2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36679 } }
I20260809 04:35:25.040346 26607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:25.041347 26613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58aa6bf3347e4c67bd6ab657353a30f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58aa6bf3347e4c67bd6ab657353a30f2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36679 } } }
I20260809 04:35:25.041512 26613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:25.041769 26614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58aa6bf3347e4c67bd6ab657353a30f2. Latest consensus state: current_term: 1 leader_uuid: "58aa6bf3347e4c67bd6ab657353a30f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58aa6bf3347e4c67bd6ab657353a30f2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36679 } } }
I20260809 04:35:25.041874 26614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:25.041860 26623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:25.044725 26623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:25.050323 26623 catalog_manager.cc:1347] Generated new cluster ID: 86d1c63d775846d5a1542528afda7287
I20260809 04:35:25.050386 26623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:25.067677 26623 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:25.068038 26623 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:25.075559 26623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2: Generated new TSK 0
I20260809 04:35:25.076316 26623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:35:25.080475 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36679
--builtin_ntp_servers=127.19.92.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 04:35:25.199007 26640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:25.199328 26640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:25.199376 26640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:25.203442 26640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:25.203500 26640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:25.203558 26640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:25.208421 26640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:25.209669 26640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:25.211066 26640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:25.220561 26652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:25.220659 26640 server_base.cc:1034] running on GCE node
W20260809 04:35:25.220548 26649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:25.220541 26648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:25.221308 26640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:25.221975 26640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:25.223141 26640 hybrid_clock.cc:648] HybridClock initialized: now 1786250125223112 us; error 44 us; skew 500 ppm
I20260809 04:35:25.224926 26640 webserver.cc:460] Webserver started at http://127.19.92.1:45127/ using document root <none> and password file <none>
I20260809 04:35:25.225559 26640 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:25.225629 26640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:25.225838 26640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:25.227890 26640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "bc053b17bcf14815a114ee18b916fd82"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "868b2a9ad27f75a96f12d2b179e1e7b3"
server_key_iv: "6e63c6243509da12a27873c29cafe389"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.228468 26640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "bc053b17bcf14815a114ee18b916fd82"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "868b2a9ad27f75a96f12d2b179e1e7b3"
server_key_iv: "6e63c6243509da12a27873c29cafe389"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.232465 26640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:35:25.235219 26661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.236538 26640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:25.236653 26640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "bc053b17bcf14815a114ee18b916fd82"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "868b2a9ad27f75a96f12d2b179e1e7b3"
server_key_iv: "6e63c6243509da12a27873c29cafe389"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.236763 26640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:25.241580 26640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:25.242208 26640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:25.242367 26640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:25.242923 26640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:25.243817 26640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:25.243865 26640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.243914 26640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:25.243943 26640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.252046 26640 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:43505
I20260809 04:35:25.252064 26831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:43505 every 8 connection(s)
I20260809 04:35:25.252889 26640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:25.258785 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.254653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:36631 (local address 127.19.92.62:36679)
0809 04:35:25.254775 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.254778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.255827 (+ 1049us) server_negotiation.cc:407] Connection header received
0809 04:35:25.257225 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.257229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.257277 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.257353 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.258354 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.258358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.258361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.258375 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.258396 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.258405 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.258483 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.258606 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.258653 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:35:25.259603 26835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.255036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36679 (local address 127.19.92.1:36631)
0809 04:35:25.255678 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.255717 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:25.256989 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:25.257532 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:25.257539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:25.257627 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:25.258106 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:35:25.258107 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:25.258145 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:35:25.258212 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:25.258473 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:25.258481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:25.258487 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:35:25.258642 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 04:35:25.258882 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":123,"threads_started":1}
I20260809 04:35:25.260303 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 26640
I20260809 04:35:25.260411 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:25.260725 19824 external_mini_cluster.cc:1428] Setting key aca100b0f8555f834538f89b53cbcd99
I20260809 04:35:25.260844 26833 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36679
I20260809 04:35:25.261068 26833 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:25.261554 26833 heartbeater.cc:511] Master 127.19.92.62:36679 requested a full tablet report, sending...
I20260809 04:35:25.262753 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36679
--builtin_ntp_servers=127.19.92.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 04:35:25.263091 26556 ts_manager.cc:194] Registered new tserver with Master: bc053b17bcf14815a114ee18b916fd82 (127.19.92.1:43505)
W20260809 04:35:25.376580 26837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:25.376843 26837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:25.376883 26837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:25.380882 26837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:25.380945 26837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:25.381000 26837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:25.386142 26837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:25.387248 26837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:25.388788 26837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:25.398787 26845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:25.398854 26837 server_base.cc:1034] running on GCE node
W20260809 04:35:25.398789 26846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:25.398789 26850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:25.399420 26837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:25.400039 26837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:25.401198 26837 hybrid_clock.cc:648] HybridClock initialized: now 1786250125401170 us; error 39 us; skew 500 ppm
I20260809 04:35:25.402971 26837 webserver.cc:460] Webserver started at http://127.19.92.2:34459/ using document root <none> and password file <none>
I20260809 04:35:25.403532 26837 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:25.403594 26837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:25.403795 26837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:25.405864 26837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "060a9f5a3517458ab4d146aabc7e210e"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "2bad7fead2d57e3a1974b140e579d956"
server_key_iv: "4de761d5cfe8e960e2b2100ac0e8c176"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.406378 26837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "060a9f5a3517458ab4d146aabc7e210e"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "2bad7fead2d57e3a1974b140e579d956"
server_key_iv: "4de761d5cfe8e960e2b2100ac0e8c176"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.409965 26837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:25.412313 26857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.413499 26837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:25.413604 26837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "060a9f5a3517458ab4d146aabc7e210e"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "2bad7fead2d57e3a1974b140e579d956"
server_key_iv: "4de761d5cfe8e960e2b2100ac0e8c176"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.413699 26837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:25.418251 26837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:25.418807 26837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:25.418951 26837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:25.419440 26837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:25.420292 26837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:25.420337 26837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.420387 26837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:25.420416 26837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.428488 26837 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:41703
I20260809 04:35:25.428582 27033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:41703 every 8 connection(s)
I20260809 04:35:25.429421 26837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:25.430006 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 26837
I20260809 04:35:25.430141 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:25.430442 19824 external_mini_cluster.cc:1428] Setting key 018755c0f8ff5410335e9b6acf53f37c
I20260809 04:35:25.432726 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36679
--builtin_ntp_servers=127.19.92.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 04:35:25.436299 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.431305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:53829 (local address 127.19.92.62:36679)
0809 04:35:25.431421 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.431424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.432906 (+ 1482us) server_negotiation.cc:407] Connection header received
0809 04:35:25.434623 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.434627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.434675 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.434767 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.435879 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.435883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.435886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.435900 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.435926 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.435936 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.436040 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.436127 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.436170 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 04:35:25.437222 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.431907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36679 (local address 127.19.92.2:53829)
0809 04:35:25.432747 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.432805 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:25.434357 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:25.434918 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:25.434926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:25.435053 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:25.435617 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:35:25.435618 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:25.435659 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:35:25.435729 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:25.436017 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:25.436022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:25.436026 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:35:25.436150 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 04:35:25.436338 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":143,"threads_started":1}
I20260809 04:35:25.438305 27034 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36679
I20260809 04:35:25.438525 27034 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:25.438922 27034 heartbeater.cc:511] Master 127.19.92.62:36679 requested a full tablet report, sending...
I20260809 04:35:25.439989 26556 ts_manager.cc:194] Registered new tserver with Master: 060a9f5a3517458ab4d146aabc7e210e (127.19.92.2:41703)
W20260809 04:35:25.550249 27038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:25.550539 27038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:25.550580 27038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:25.554605 27038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:25.554663 27038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:25.554723 27038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:25.559506 27038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:25.560599 27038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:25.562039 27038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:25.571419 27048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:25.571472 27038 server_base.cc:1034] running on GCE node
W20260809 04:35:25.571419 27044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:25.571399 27045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:25.572063 27038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:25.572687 27038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:25.573849 27038 hybrid_clock.cc:648] HybridClock initialized: now 1786250125573828 us; error 35 us; skew 500 ppm
I20260809 04:35:25.575832 27038 webserver.cc:460] Webserver started at http://127.19.92.3:38495/ using document root <none> and password file <none>
I20260809 04:35:25.576408 27038 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:25.576476 27038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:25.576704 27038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:25.579034 27038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "5dde848ce3254e64b9c97177c88e806b"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "c47c076d085509e82beba8478eb5de19"
server_key_iv: "cc9961bf12ce4b674d16e23affa746f3"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.579612 27038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "5dde848ce3254e64b9c97177c88e806b"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "c47c076d085509e82beba8478eb5de19"
server_key_iv: "cc9961bf12ce4b674d16e23affa746f3"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.583371 27038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 04:35:25.585990 27058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.587186 27038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:25.587296 27038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "5dde848ce3254e64b9c97177c88e806b"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "c47c076d085509e82beba8478eb5de19"
server_key_iv: "cc9961bf12ce4b674d16e23affa746f3"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.587400 27038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:25.592293 27038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:25.592970 27038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:25.593187 27038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:25.593772 27038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:25.594725 27038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:25.594774 27038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.594825 27038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:25.594854 27038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.603160 27038 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:42897
I20260809 04:35:25.603193 27222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:42897 every 8 connection(s)
I20260809 04:35:25.604059 27038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:25.610136 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.605961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:51705 (local address 127.19.92.62:36679)
0809 04:35:25.606096 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.606100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.607166 (+ 1066us) server_negotiation.cc:407] Connection header received
0809 04:35:25.608480 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.608484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.608531 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.608607 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.609698 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.609701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.609704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.609716 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.609737 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.609744 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.609822 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.609944 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.609988 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:35:25.610528 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 27038
I20260809 04:35:25.610633 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:25.610886 19824 external_mini_cluster.cc:1428] Setting key ee562d47227f23c201c1826da49ff433
I20260809 04:35:25.611016 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.606358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36679 (local address 127.19.92.3:51705)
0809 04:35:25.607021 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.607060 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:25.608267 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:25.608784 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:25.608792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:25.608899 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:25.609436 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:35:25.609437 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:25.609480 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:35:25.609556 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:25.609813 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:25.609821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:25.609827 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:35:25.609984 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 04:35:25.610223 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260809 04:35:25.612138 27223 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36679
I20260809 04:35:25.612356 27223 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:25.612877 27223 heartbeater.cc:511] Master 127.19.92.62:36679 requested a full tablet report, sending...
I20260809 04:35:25.613933 26554 ts_manager.cc:194] Registered new tserver with Master: 5dde848ce3254e64b9c97177c88e806b (127.19.92.3:42897)
I20260809 04:35:25.624671 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:35:25.628155 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.626619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.19.92.62:36679)
0809 04:35:25.626734 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.626738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.626905 (+ 167us) server_negotiation.cc:407] Connection header received
0809 04:35:25.627033 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.627036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.627074 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.627137 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.627778 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.627782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.627785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.627796 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.627817 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.627825 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.627892 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.627974 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.628021 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 04:35:25.631091 26554 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:25.634307 26554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:25.645862 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.642774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:42897 (local address 127.0.0.1:47298)
0809 04:35:25.643529 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.643580 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:25.643694 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:25.644244 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:25.644251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:25.644332 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:25.645049 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:35:25.645051 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:25.645072 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:35:25.645142 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:25.645519 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:25.645530 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:25.645537 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:35:25.645649 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 04:35:25.645719 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":85,"threads_started":1}
I20260809 04:35:25.645903 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.643678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.19.92.2:41703)
0809 04:35:25.644012 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.644031 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.644426 (+ 395us) server_negotiation.cc:407] Connection header received
0809 04:35:25.644482 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.644488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.644667 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.644785 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.645411 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.645416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.645425 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.645434 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.645467 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.645490 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.645616 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.645693 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.645764 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":136,"threads_started":1}
I20260809 04:35:25.646291 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.642959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.19.92.3:42897)
0809 04:35:25.643360 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.643374 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.643810 (+ 436us) server_negotiation.cc:407] Connection header received
0809 04:35:25.643876 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.643881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.644090 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.644231 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.645309 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.645315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.645327 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.645339 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.645386 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.645419 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.645541 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.646063 (+ 522us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.646141 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":174,"threads_started":1}
I20260809 04:35:25.646384 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.643498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:41703 (local address 127.0.0.1:57882)
0809 04:35:25.644245 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.644257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:25.644328 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:25.644816 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:25.644819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:25.644834 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:25.645245 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:35:25.645247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:25.645258 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:35:25.645287 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:25.645600 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:25.645603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:25.645607 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:35:25.645682 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 04:35:25.645727 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260809 04:35:25.648286 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.644228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.19.92.1:43505)
0809 04:35:25.645012 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.645024 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.645544 (+ 520us) server_negotiation.cc:407] Connection header received
0809 04:35:25.645712 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.645718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.645886 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.646326 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.647719 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.647725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.647737 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.647749 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.647793 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.647828 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.647902 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.648078 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.648142 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":145,"threads_started":1}
I20260809 04:35:25.648644 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.643822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:43505 (local address 127.0.0.1:39424)
0809 04:35:25.645450 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.645459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:25.645601 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:25.647104 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:25.647107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:25.647121 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:25.647547 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:35:25.647549 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:25.647560 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:35:25.647585 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:25.647972 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:25.647975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:25.647977 (+ 2us) client_negotiation.cc:769] Sending connection context
0809 04:35:25.648498 (+ 521us) client_negotiation.cc:240] Negotiation successful
0809 04:35:25.648535 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572,"thread_start_us":92,"threads_started":1}
I20260809 04:35:25.649171 26931 tablet_service.cc:1467] Processing CreateTablet for tablet 4824855fe23041d4a1b08e8e86983f3b (DEFAULT_TABLE table=test-table [id=c7e9b56b7adf4600be60aa88d1ed089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:25.649186 27134 tablet_service.cc:1467] Processing CreateTablet for tablet 4824855fe23041d4a1b08e8e86983f3b (DEFAULT_TABLE table=test-table [id=c7e9b56b7adf4600be60aa88d1ed089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:25.650108 26931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4824855fe23041d4a1b08e8e86983f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:25.650110 27134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4824855fe23041d4a1b08e8e86983f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:25.655810 27253 tablet_bootstrap.cc:492] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: Bootstrap starting.
I20260809 04:35:25.655989 26734 tablet_service.cc:1467] Processing CreateTablet for tablet 4824855fe23041d4a1b08e8e86983f3b (DEFAULT_TABLE table=test-table [id=c7e9b56b7adf4600be60aa88d1ed089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:25.657008 26734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4824855fe23041d4a1b08e8e86983f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:25.657666 27255 tablet_bootstrap.cc:492] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Bootstrap starting.
I20260809 04:35:25.657892 27253 tablet_bootstrap.cc:654] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:25.658555 27253 log.cc:824] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:25.659665 27255 tablet_bootstrap.cc:654] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:25.660399 27253 tablet_bootstrap.cc:492] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: No bootstrap required, opened a new log
I20260809 04:35:25.660501 27255 log.cc:824] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:25.660583 27253 ts_tablet_manager.cc:1397] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 04:35:25.662416 27255 tablet_bootstrap.cc:492] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: No bootstrap required, opened a new log
I20260809 04:35:25.662642 27255 ts_tablet_manager.cc:1397] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:35:25.664211 27253 raft_consensus.cc:348] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.664597 27253 raft_consensus.cc:374] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:25.664611 27259 tablet_bootstrap.cc:492] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82: Bootstrap starting.
I20260809 04:35:25.664667 27253 raft_consensus.cc:729] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dde848ce3254e64b9c97177c88e806b, State: Initialized, Role: FOLLOWER
I20260809 04:35:25.665122 27253 consensus_queue.cc:260] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.665776 27223 heartbeater.cc:503] Master 127.19.92.62:36679 was elected leader, sending a full tablet report...
I20260809 04:35:25.666131 27253 ts_tablet_manager.cc:1428] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:35:25.666726 27259 tablet_bootstrap.cc:654] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:25.667021 27255 raft_consensus.cc:348] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.667277 27255 raft_consensus.cc:374] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:25.667373 27255 raft_consensus.cc:729] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060a9f5a3517458ab4d146aabc7e210e, State: Initialized, Role: FOLLOWER
I20260809 04:35:25.667580 27259 log.cc:824] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:25.667850 27255 consensus_queue.cc:260] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.668512 27034 heartbeater.cc:503] Master 127.19.92.62:36679 was elected leader, sending a full tablet report...
I20260809 04:35:25.668833 27255 ts_tablet_manager.cc:1428] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:35:25.670542 27259 tablet_bootstrap.cc:492] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82: No bootstrap required, opened a new log
I20260809 04:35:25.670747 27259 ts_tablet_manager.cc:1397] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:35:25.674489 27259 raft_consensus.cc:348] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.674718 27259 raft_consensus.cc:374] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:25.674754 27259 raft_consensus.cc:729] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc053b17bcf14815a114ee18b916fd82, State: Initialized, Role: FOLLOWER
I20260809 04:35:25.675161 27259 consensus_queue.cc:260] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.675801 26833 heartbeater.cc:503] Master 127.19.92.62:36679 was elected leader, sending a full tablet report...
I20260809 04:35:25.676023 27259 ts_tablet_manager.cc:1428] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260809 04:35:25.680316 27036 tablet.cc:2369] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:35:25.706620 27260 raft_consensus.cc:482] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:25.706833 27260 raft_consensus.cc:504] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.708176 27260 leader_election.cc:288] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060a9f5a3517458ab4d146aabc7e210e (127.19.92.2:41703), bc053b17bcf14815a114ee18b916fd82 (127.19.92.1:43505)
I20260809 04:35:25.710079 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.708530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:57703 (local address 127.19.92.2:41703)
0809 04:35:25.708694 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.708699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.708714 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:35:25.708762 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.708765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.708804 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.708886 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.709689 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.709692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.709694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.709705 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.709727 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.709739 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.709831 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.709923 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.709968 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:35:25.710083 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.708383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:41703 (local address 127.19.92.3:57703)
0809 04:35:25.708548 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.708565 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:25.708650 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:25.709001 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:25.709004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:25.709017 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:25.709536 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:35:25.709538 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:25.709548 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:35:25.709577 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:25.709827 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:25.709830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:25.709834 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:35:25.709911 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:35:25.709959 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 04:35:25.710373 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.708652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:43505 (local address 127.19.92.3:52053)
0809 04:35:25.709016 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.709028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:25.709135 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:25.709429 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:25.709432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:25.709447 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:25.709830 (+ 383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:35:25.709832 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:25.709843 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:35:25.709870 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:25.710108 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:25.710112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:25.710115 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 04:35:25.710192 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:35:25.710227 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":137,"threads_started":1}
I20260809 04:35:25.710469 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.708780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:52053 (local address 127.19.92.1:43505)
0809 04:35:25.708909 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.708913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.709132 (+ 219us) server_negotiation.cc:407] Connection header received
0809 04:35:25.709290 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.709293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.709332 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.709394 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.709977 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.709981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.709983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.709996 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.710018 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.710027 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.710114 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.710286 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.710334 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:25.710807 26965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4824855fe23041d4a1b08e8e86983f3b" candidate_uuid: "5dde848ce3254e64b9c97177c88e806b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060a9f5a3517458ab4d146aabc7e210e" is_pre_election: true
I20260809 04:35:25.711122 26965 raft_consensus.cc:2454] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dde848ce3254e64b9c97177c88e806b in term 0.
I20260809 04:35:25.711601 27061 leader_election.cc:302] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060a9f5a3517458ab4d146aabc7e210e, 5dde848ce3254e64b9c97177c88e806b; no voters:
I20260809 04:35:25.711917 27260 raft_consensus.cc:2781] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:25.711926 26767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4824855fe23041d4a1b08e8e86983f3b" candidate_uuid: "5dde848ce3254e64b9c97177c88e806b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc053b17bcf14815a114ee18b916fd82" is_pre_election: true
I20260809 04:35:25.712020 27260 raft_consensus.cc:482] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:25.712078 27260 raft_consensus.cc:3037] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:25.712210 26767 raft_consensus.cc:2454] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dde848ce3254e64b9c97177c88e806b in term 0.
I20260809 04:35:25.713274 27260 raft_consensus.cc:504] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.713595 27260 leader_election.cc:288] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [CANDIDATE]: Term 1 election: Requested vote from peers 060a9f5a3517458ab4d146aabc7e210e (127.19.92.2:41703), bc053b17bcf14815a114ee18b916fd82 (127.19.92.1:43505)
I20260809 04:35:25.713923 26965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4824855fe23041d4a1b08e8e86983f3b" candidate_uuid: "5dde848ce3254e64b9c97177c88e806b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060a9f5a3517458ab4d146aabc7e210e"
I20260809 04:35:25.713985 26767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4824855fe23041d4a1b08e8e86983f3b" candidate_uuid: "5dde848ce3254e64b9c97177c88e806b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc053b17bcf14815a114ee18b916fd82"
I20260809 04:35:25.714085 26965 raft_consensus.cc:3037] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:25.714098 26767 raft_consensus.cc:3037] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:25.715407 26965 raft_consensus.cc:2454] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dde848ce3254e64b9c97177c88e806b in term 1.
I20260809 04:35:25.715407 26767 raft_consensus.cc:2454] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dde848ce3254e64b9c97177c88e806b in term 1.
I20260809 04:35:25.715677 27061 leader_election.cc:302] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060a9f5a3517458ab4d146aabc7e210e, 5dde848ce3254e64b9c97177c88e806b; no voters:
I20260809 04:35:25.715837 27260 raft_consensus.cc:2781] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:25.716068 27260 raft_consensus.cc:686] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 1 LEADER]: Becoming Leader. State: Replica: 5dde848ce3254e64b9c97177c88e806b, State: Running, Role: LEADER
I20260809 04:35:25.716425 27260 consensus_queue.cc:237] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } }
I20260809 04:35:25.719705 26553 catalog_manager.cc:5495] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dde848ce3254e64b9c97177c88e806b (127.19.92.3). New cstate: current_term: 1 leader_uuid: "5dde848ce3254e64b9c97177c88e806b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dde848ce3254e64b9c97177c88e806b" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 } health_report { overall_health: UNKNOWN } } }
I20260809 04:35:25.729194 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.727954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.19.92.3:42897)
0809 04:35:25.728061 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:25.728064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:25.728076 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:35:25.728178 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:25.728180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:25.728244 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:25.728315 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:25.728865 (+ 550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:25.728868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:25.728870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:25.728881 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:25.728901 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:25.728909 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:25.728981 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:25.729029 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 04:35:25.729062 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 04:35:25.734876 26767 raft_consensus.cc:1261] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [term 1 FOLLOWER]: Refusing update from remote peer 5dde848ce3254e64b9c97177c88e806b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:25.734889 26965 raft_consensus.cc:1261] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 1 FOLLOWER]: Refusing update from remote peer 5dde848ce3254e64b9c97177c88e806b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:25.735558 27260 consensus_queue.cc:1035] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [LEADER]: Connected to new peer: Peer: permanent_uuid: "060a9f5a3517458ab4d146aabc7e210e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:25.735770 27267 consensus_queue.cc:1035] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc053b17bcf14815a114ee18b916fd82" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:25.745273 27274 mvcc.cc:205] Tried to move back new op lower bound from 7316480515003187200 to 7316480514937708544. Current Snapshot: MvccSnapshot[applied={T|T < 7316480515003187200 or (T in {7316480515003187200})}]
I20260809 04:35:25.753775 26556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:25.754005 26556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:25.756875 26556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58aa6bf3347e4c67bd6ab657353a30f2: Sending DeleteTablet for 3 replicas of tablet 4824855fe23041d4a1b08e8e86983f3b
I20260809 04:35:25.757895 26931 tablet_service.cc:1514] Processing DeleteTablet for tablet 4824855fe23041d4a1b08e8e86983f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:25 UTC) from {username='slave'} at 127.0.0.1:57882
I20260809 04:35:25.757916 26734 tablet_service.cc:1514] Processing DeleteTablet for tablet 4824855fe23041d4a1b08e8e86983f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:25 UTC) from {username='slave'} at 127.0.0.1:39424
I20260809 04:35:25.758064 27134 tablet_service.cc:1514] Processing DeleteTablet for tablet 4824855fe23041d4a1b08e8e86983f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:25 UTC) from {username='slave'} at 127.0.0.1:47298
I20260809 04:35:25.758484 27292 tablet_replica.cc:330] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82: stopping tablet replica
I20260809 04:35:25.758760 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 26640
I20260809 04:35:25.758863 27292 raft_consensus.cc:2229] T 4824855fe23041d4a1b08e8e86983f3b P bc053b17bcf14815a114ee18b916fd82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:25.758888 27298 tablet_replica.cc:330] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: stopping tablet replica
I20260809 04:35:25.759590 27298 raft_consensus.cc:2229] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:25.760108 27298 raft_consensus.cc:2258] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:25.764657 27298 ts_tablet_manager.cc:1905] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:25.767704 27298 ts_tablet_manager.cc:1918] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:25.767812 27298 log.cc:1196] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/4824855fe23041d4a1b08e8e86983f3b
I20260809 04:35:25.768122 27298 ts_tablet_manager.cc:1939] T 4824855fe23041d4a1b08e8e86983f3b P 5dde848ce3254e64b9c97177c88e806b: Deleting consensus metadata
I20260809 04:35:25.769644 27293 tablet_replica.cc:330] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: stopping tablet replica
I20260809 04:35:25.770081 27293 raft_consensus.cc:2229] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:25.770530 27293 raft_consensus.cc:2258] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:25.770865 26527 catalog_manager.cc:4841] TS 5dde848ce3254e64b9c97177c88e806b (127.19.92.3:42897): tablet 4824855fe23041d4a1b08e8e86983f3b (table test-table [id=c7e9b56b7adf4600be60aa88d1ed089f]) successfully deleted
I20260809 04:35:25.772922 27293 ts_tablet_manager.cc:1905] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:25.775640 27293 ts_tablet_manager.cc:1918] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:25.775753 27293 log.cc:1196] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/4824855fe23041d4a1b08e8e86983f3b
I20260809 04:35:25.776122 27293 ts_tablet_manager.cc:1939] T 4824855fe23041d4a1b08e8e86983f3b P 060a9f5a3517458ab4d146aabc7e210e: Deleting consensus metadata
I20260809 04:35:25.777498 26527 catalog_manager.cc:4841] TS 060a9f5a3517458ab4d146aabc7e210e (127.19.92.2:41703): tablet 4824855fe23041d4a1b08e8e86983f3b (table test-table [id=c7e9b56b7adf4600be60aa88d1ed089f]) successfully deleted
W20260809 04:35:25.778357 26527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:43505 (error 108)
I20260809 04:35:25.779136 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 26837
I20260809 04:35:25.779414 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:25.779135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:43505 (local address 127.0.0.1:39438)
0809 04:35:25.779267 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:25.779334 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:43505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 04:35:25.781374 26527 catalog_manager.cc:4584] TS bc053b17bcf14815a114ee18b916fd82 (127.19.92.1:43505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4824855fe23041d4a1b08e8e86983f3b: Network error: Client connection negotiation failed: client connection to 127.19.92.1:43505: connect: Connection refused (error 111)
I20260809 04:35:25.790719 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 27038
I20260809 04:35:25.810443 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 26495
2026-08-09T04:35:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (987 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 04:35:25.836273 19824 mini_oidc.cc:152] Starting JWKS server
I20260809 04:35:25.838104 19824 webserver.cc:460] Webserver started at https://127.0.0.1:34349/ using document root <none> and password file <none>
I20260809 04:35:25.838274 19824 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 04:35:25.838539 19824 webserver.cc:460] Webserver started at http://127.0.0.1:33435/ using document root <none> and password file <none>
2026-08-09T04:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:25Z Disabled control of system clock
I20260809 04:35:25.848378 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:39087
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:39087
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34349/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/testchainca.cert with env {}
W20260809 04:35:25.961687 27307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:25.961975 27307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:25.962019 27307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:25.966109 27307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:25.966173 27307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:25.966187 27307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:25.966202 27307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:25.966216 27307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 04:35:25.966234 27307 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 04:35:25.966262 27307 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34349/jwks.json
I20260809 04:35:25.971594 27307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:39087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34349/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:25.972666 27307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:25.973731 27307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:25.981671 27318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:25.981679 27316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:25.981841 27307 server_base.cc:1034] running on GCE node
W20260809 04:35:25.981700 27314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:25.982498 27307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:25.983572 27307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:25.984738 27307 hybrid_clock.cc:648] HybridClock initialized: now 1786250125984712 us; error 39 us; skew 500 ppm
I20260809 04:35:25.986569 27307 webserver.cc:460] Webserver started at http://127.19.92.62:38699/ using document root <none> and password file <none>
I20260809 04:35:25.987123 27307 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:25.987185 27307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:25.987385 27307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:25.989495 27307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "695da2178e4f4332a4f32345d1f45c5f"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "843c08d7c89dde6c476de80bd2db6959"
server_key_iv: "2c98146c68b6519f37fb4e9c6fc26ab1"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.990059 27307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "695da2178e4f4332a4f32345d1f45c5f"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "843c08d7c89dde6c476de80bd2db6959"
server_key_iv: "2c98146c68b6519f37fb4e9c6fc26ab1"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.993952 27307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:35:25.996345 27326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:25.997628 27307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:35:25.997756 27307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "695da2178e4f4332a4f32345d1f45c5f"
format_stamp: "Formatted at 2026-08-09 04:35:25 on dist-test-slave-rmhg"
server_key: "843c08d7c89dde6c476de80bd2db6959"
server_key_iv: "2c98146c68b6519f37fb4e9c6fc26ab1"
server_key_version: "encryptionkey@0"
I20260809 04:35:25.997852 27307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:26.013046 27307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:26.013765 27307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:26.013938 27307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:26.020754 27307 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:39087
I20260809 04:35:26.020772 27402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:39087 every 8 connection(s)
I20260809 04:35:26.021720 27307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:26.024394 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 27307
I20260809 04:35:26.024525 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:26.024794 19824 external_mini_cluster.cc:1428] Setting key ae1622fde2b7f4466d47c221f8f14373
I20260809 04:35:26.024830 27403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:26.031844 27403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f: Bootstrap starting.
I20260809 04:35:26.034214 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:26.026124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.19.92.62:39087)
0809 04:35:26.026728 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:26.026744 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:26.026779 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:35:26.027482 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:26.027501 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:26.027889 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:26.028164 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:26.028511 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.029538 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.030784 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.031299 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.031431 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.032361 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:26.032366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:26.032380 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:26.032809 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:26.032870 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:26.032892 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:26.032983 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:26.033146 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 04:35:26.033451 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260809 04:35:26.034322 27403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:26.035068 27403 log.cc:824] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:26.037122 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:35:26.037369 27403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f: No bootstrap required, opened a new log
I20260809 04:35:26.041272 27403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695da2178e4f4332a4f32345d1f45c5f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39087 } }
I20260809 04:35:26.041505 27403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:26.041558 27403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695da2178e4f4332a4f32345d1f45c5f, State: Initialized, Role: FOLLOWER
I20260809 04:35:26.041925 27403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695da2178e4f4332a4f32345d1f45c5f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39087 } }
I20260809 04:35:26.042047 27403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:26.042110 27403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:26.042186 27403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:26.043021 27403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695da2178e4f4332a4f32345d1f45c5f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39087 } }
I20260809 04:35:26.043366 27403 leader_election.cc:302] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695da2178e4f4332a4f32345d1f45c5f; no voters:
I20260809 04:35:26.043707 27403 leader_election.cc:288] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:26.043812 27412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:26.043987 27412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [term 1 LEADER]: Becoming Leader. State: Replica: 695da2178e4f4332a4f32345d1f45c5f, State: Running, Role: LEADER
I20260809 04:35:26.044312 27412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695da2178e4f4332a4f32345d1f45c5f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39087 } }
I20260809 04:35:26.045027 27403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:26.046051 27413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695da2178e4f4332a4f32345d1f45c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695da2178e4f4332a4f32345d1f45c5f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39087 } } }
I20260809 04:35:26.046185 27413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:26.046509 27415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695da2178e4f4332a4f32345d1f45c5f. Latest consensus state: current_term: 1 leader_uuid: "695da2178e4f4332a4f32345d1f45c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695da2178e4f4332a4f32345d1f45c5f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 39087 } } }
I20260809 04:35:26.046594 27424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:26.046680 27415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:26.049872 27424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:26.055150 27424 catalog_manager.cc:1347] Generated new cluster ID: e867acfb805b4acaa52ba853f4ba2c3a
I20260809 04:35:26.055207 27424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:26.071780 27424 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:26.073132 27424 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:26.083362 27424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695da2178e4f4332a4f32345d1f45c5f: Generated new TSK 0
I20260809 04:35:26.084237 27424 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 04:35:26.122529 27461 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 04:35:26.122707 27461 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260809 04:35:26.124271 27455 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:26.096668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:39087 (local address 127.0.0.1:56990)
0809 04:35:26.097024 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:26.097039 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:26.097199 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:26.097513 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:26.097516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:26.097526 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 04:35:26.097663 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:26.097669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.098351 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.098355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:26.099372 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:26.099384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.099775 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.099779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:26.099873 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.099939 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 04:35:26.122887 (+ 22948us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 04:35:26.123098 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.92.62:39087: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.92.62:39087: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 04:35:26.124848 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:26.096754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.19.92.62:39087)
0809 04:35:26.096882 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:26.096887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:26.097107 (+ 220us) server_negotiation.cc:407] Connection header received
0809 04:35:26.097348 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:26.097351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:26.097398 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:26.097479 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0809 04:35:26.097755 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.098263 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.099480 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.099685 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.099750 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.100101 (+ 351us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 04:35:26.122598 (+ 22497us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 04:35:26.124701 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 04:35:26.124863 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 27307
W20260809 04:35:26.124924 27409 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
2026-08-09T04:35:26Z chronyd exiting
I20260809 04:35:26.167835 19824 test_util.cc:175] -----------------------------------------------
I20260809 04:35:26.167922 19824 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250097044341-19824-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 04:35:26.168982 19824 mini_oidc.cc:152] Starting JWKS server
I20260809 04:35:26.169860 19824 webserver.cc:460] Webserver started at https://127.0.0.1:36097/ using document root <none> and password file <none>
I20260809 04:35:26.169943 19824 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 04:35:26.170157 19824 webserver.cc:460] Webserver started at http://127.0.0.1:43693/ using document root <none> and password file <none>
2026-08-09T04:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:26Z Disabled control of system clock
I20260809 04:35:26.179960 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:44031
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:44031
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/testchainca.cert with env {}
W20260809 04:35:26.291590 27473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:26.291879 27473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:26.291920 27473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:26.296003 27473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:26.296069 27473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:26.296083 27473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:26.296099 27473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:26.296113 27473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 04:35:26.296128 27473 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 04:35:26.296139 27473 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36097/jwks.json
I20260809 04:35:26.301638 27473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:44031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:44031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:26.302707 27473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:26.303699 27473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:26.312425 27484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:26.312428 27478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:26.312429 27481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:26.312856 27473 server_base.cc:1034] running on GCE node
I20260809 04:35:26.313548 27473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:26.314826 27473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:26.315996 27473 hybrid_clock.cc:648] HybridClock initialized: now 1786250126315985 us; error 31 us; skew 500 ppm
I20260809 04:35:26.320977 27473 webserver.cc:460] Webserver started at http://127.19.92.62:38963/ using document root <none> and password file <none>
I20260809 04:35:26.321657 27473 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:26.321736 27473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:26.321954 27473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:26.324156 27473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "3b128884b81a42c78be928998753078d"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "ce74e0708b1603e220d048716eacf22e"
server_key_iv: "cfc37a513ac48a56f0e1104801fc0685"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.324764 27473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "3b128884b81a42c78be928998753078d"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "ce74e0708b1603e220d048716eacf22e"
server_key_iv: "cfc37a513ac48a56f0e1104801fc0685"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.328665 27473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:35:26.331308 27490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:26.332630 27473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:35:26.332756 27473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "3b128884b81a42c78be928998753078d"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "ce74e0708b1603e220d048716eacf22e"
server_key_iv: "cfc37a513ac48a56f0e1104801fc0685"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.332866 27473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:26.352089 27473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:26.352886 27473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:26.353096 27473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:26.365515 27473 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:44031
I20260809 04:35:26.365542 27572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:44031 every 8 connection(s)
I20260809 04:35:26.366873 27473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:26.368371 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 27473
I20260809 04:35:26.368479 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:26.368768 19824 external_mini_cluster.cc:1428] Setting key e45eca5aa13c29c80afa625b4486d804
I20260809 04:35:26.370990 27573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:26.378088 27573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d: Bootstrap starting.
I20260809 04:35:26.381173 27573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:26.382115 27573 log.cc:824] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:26.383829 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:26.370347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.19.92.62:44031)
0809 04:35:26.371186 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:26.371210 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:26.371258 (+ 48us) server_negotiation.cc:407] Connection header received
0809 04:35:26.371983 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:26.372001 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:26.372510 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:26.372898 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:26.373440 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.375017 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.376339 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.377102 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.377291 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.381640 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:26.381647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:26.381665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:26.382194 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:26.382272 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:26.382302 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:26.382405 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:26.382571 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 04:35:26.382874 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":139,"threads_started":1}
I20260809 04:35:26.384588 27573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d: No bootstrap required, opened a new log
I20260809 04:35:26.386858 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:35:26.390244 27573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b128884b81a42c78be928998753078d" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44031 } }
I20260809 04:35:26.390619 27573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:26.390690 27573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b128884b81a42c78be928998753078d, State: Initialized, Role: FOLLOWER
I20260809 04:35:26.391249 27573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b128884b81a42c78be928998753078d" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44031 } }
I20260809 04:35:26.391433 27573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:26.391517 27573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:26.391634 27573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:26.393035 27573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b128884b81a42c78be928998753078d" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44031 } }
I20260809 04:35:26.393503 27573 leader_election.cc:302] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b128884b81a42c78be928998753078d; no voters:
I20260809 04:35:26.393932 27573 leader_election.cc:288] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:26.394079 27583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:26.394374 27583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [term 1 LEADER]: Becoming Leader. State: Replica: 3b128884b81a42c78be928998753078d, State: Running, Role: LEADER
I20260809 04:35:26.394764 27583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b128884b81a42c78be928998753078d" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44031 } }
I20260809 04:35:26.395479 27573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:26.396653 27584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b128884b81a42c78be928998753078d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b128884b81a42c78be928998753078d" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44031 } } }
I20260809 04:35:26.396795 27584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:26.397068 27585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b128884b81a42c78be928998753078d. Latest consensus state: current_term: 1 leader_uuid: "3b128884b81a42c78be928998753078d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b128884b81a42c78be928998753078d" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44031 } } }
I20260809 04:35:26.397204 27585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:26.397289 27597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:26.401372 27597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:26.407742 27597 catalog_manager.cc:1347] Generated new cluster ID: dcf1e00cc52f4ac0b0fd11dd458a4a07
I20260809 04:35:26.407873 27597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:26.420428 27597 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:26.421466 27597 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:26.430198 27597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b128884b81a42c78be928998753078d: Generated new TSK 0
I20260809 04:35:26.431031 27597 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 04:35:26.463891 27619 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 04:35:26.463997 27619 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 04:35:26.464715 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:26.448971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.19.92.62:44031)
0809 04:35:26.449115 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:26.449119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:26.449309 (+ 190us) server_negotiation.cc:407] Connection header received
0809 04:35:26.449475 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:26.449477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:26.449526 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:26.449624 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0809 04:35:26.449857 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.450396 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.451640 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.451863 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.451958 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.452263 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 04:35:26.464337 (+ 12074us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 04:35:26.464594 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 04:35:26.464780 27579 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 04:35:26.464985 27617 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:26.448880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:44031 (local address 127.0.0.1:34488)
0809 04:35:26.449224 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:26.449239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:26.449355 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:26.449633 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:26.449636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:26.449644 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0809 04:35:26.449766 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:26.449773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.450496 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.450500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:26.451510 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:26.451521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.451976 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.451980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:26.452081 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.452112 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 04:35:26.464575 (+ 12463us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 04:35:26.464797 (+ 222us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.92.62:44031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260809 04:35:26.465734 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 27473
2026-08-09T04:35:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (344 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 04:35:26.513621 19824 mini_oidc.cc:152] Starting JWKS server
I20260809 04:35:26.514497 19824 webserver.cc:460] Webserver started at https://127.0.0.1:44029/ using document root <none> and password file <none>
I20260809 04:35:26.514595 19824 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 04:35:26.514849 19824 webserver.cc:460] Webserver started at http://127.0.0.1:38907/ using document root <none> and password file <none>
2026-08-09T04:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:26Z Disabled control of system clock
I20260809 04:35:26.525465 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:43939
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:43939
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44029/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 04:35:26.639809 27628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:26.640108 27628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:26.640154 27628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:26.644294 27628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:26.644364 27628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:26.644378 27628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:26.644394 27628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:26.644408 27628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 04:35:26.644465 27628 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 04:35:26.644490 27628 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44029/jwks.json
I20260809 04:35:26.649400 27628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:43939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44029/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:26.650566 27628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:26.651950 27628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:26.660228 27638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:26.660230 27635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:26.660248 27634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:26.660301 27628 server_base.cc:1034] running on GCE node
I20260809 04:35:26.661029 27628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:26.662122 27628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:26.663280 27628 hybrid_clock.cc:648] HybridClock initialized: now 1786250126663268 us; error 28 us; skew 500 ppm
I20260809 04:35:26.665067 27628 webserver.cc:460] Webserver started at http://127.19.92.62:45483/ using document root <none> and password file <none>
I20260809 04:35:26.665640 27628 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:26.665707 27628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:26.665900 27628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:26.667869 27628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "410fe748845f484d9c26de385163c75f"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "a6ba5d3d4e302f1644a3ac8c4c73c6f9"
server_key_iv: "954c2edaa2ff633c512a8413e85120dc"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.668403 27628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "410fe748845f484d9c26de385163c75f"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "a6ba5d3d4e302f1644a3ac8c4c73c6f9"
server_key_iv: "954c2edaa2ff633c512a8413e85120dc"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.672072 27628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:35:26.674734 27649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:26.676028 27628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:26.676156 27628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "410fe748845f484d9c26de385163c75f"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "a6ba5d3d4e302f1644a3ac8c4c73c6f9"
server_key_iv: "954c2edaa2ff633c512a8413e85120dc"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.676270 27628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:26.706287 27628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:26.707022 27628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:26.707207 27628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:26.714233 27628 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:43939
I20260809 04:35:26.714250 27737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:43939 every 8 connection(s)
I20260809 04:35:26.715308 27628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:26.718536 27738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:26.721735 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 27628
I20260809 04:35:26.721854 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:26.722114 19824 external_mini_cluster.cc:1428] Setting key 8c907717641a053c6e8986a66659ecd3
I20260809 04:35:26.724509 27738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f: Bootstrap starting.
I20260809 04:35:26.726912 27738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:26.727813 27738 log.cc:824] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:26.729839 27738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f: No bootstrap required, opened a new log
I20260809 04:35:26.731153 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:26.723392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.19.92.62:43939)
0809 04:35:26.724020 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:26.724037 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:26.724074 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:35:26.724748 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:26.724769 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:26.725233 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:26.725524 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:26.725881 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.726859 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.728043 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.728520 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.728671 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.729471 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:26.729476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:26.729498 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:26.729874 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:26.729944 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:26.729971 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:26.730058 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:26.730195 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 04:35:26.730448 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260809 04:35:26.733636 27738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410fe748845f484d9c26de385163c75f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43939 } }
I20260809 04:35:26.733913 27738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:26.733978 27738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410fe748845f484d9c26de385163c75f, State: Initialized, Role: FOLLOWER
I20260809 04:35:26.734084 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:35:26.734433 27738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410fe748845f484d9c26de385163c75f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43939 } }
I20260809 04:35:26.734563 27738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:26.734634 27738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:26.734733 27738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:26.735886 27738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410fe748845f484d9c26de385163c75f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43939 } }
I20260809 04:35:26.736217 27738 leader_election.cc:302] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 410fe748845f484d9c26de385163c75f; no voters:
I20260809 04:35:26.736519 27738 leader_election.cc:288] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:26.736635 27743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:26.736846 27743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [term 1 LEADER]: Becoming Leader. State: Replica: 410fe748845f484d9c26de385163c75f, State: Running, Role: LEADER
I20260809 04:35:26.737214 27743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410fe748845f484d9c26de385163c75f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43939 } }
I20260809 04:35:26.737834 27738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:26.738818 27747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "410fe748845f484d9c26de385163c75f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410fe748845f484d9c26de385163c75f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43939 } } }
I20260809 04:35:26.738957 27747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:26.738876 27748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 410fe748845f484d9c26de385163c75f. Latest consensus state: current_term: 1 leader_uuid: "410fe748845f484d9c26de385163c75f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410fe748845f484d9c26de385163c75f" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43939 } } }
I20260809 04:35:26.739042 27748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:26.739391 27759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:26.742473 27759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:26.747815 27759 catalog_manager.cc:1347] Generated new cluster ID: 56b7a92654bd4382a783ce810ed873ac
I20260809 04:35:26.747895 27759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:26.765846 27759 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:26.766813 27759 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:26.773809 27759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 410fe748845f484d9c26de385163c75f: Generated new TSK 0
I20260809 04:35:26.774504 27759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:35:26.788839 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:26.783838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.19.92.62:43939)
0809 04:35:26.783965 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:26.783970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:26.784146 (+ 176us) server_negotiation.cc:407] Connection header received
0809 04:35:26.784306 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:26.784308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:26.784355 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:26.784453 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0809 04:35:26.784691 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.785258 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.786488 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.786724 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.786797 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.787115 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 04:35:26.788452 (+ 1337us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 04:35:26.788689 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 04:35:26.788909 27741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 04:35:26.790756 27778 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:26.783772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43939 (local address 127.0.0.1:54800)
0809 04:35:26.784060 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:26.784076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:26.784194 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:26.784464 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:26.784467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:26.784476 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 04:35:26.784595 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:26.784602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.785359 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.785362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:26.786359 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:26.786372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:26.786832 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:26.786836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:26.786928 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:26.786958 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 04:35:26.788633 (+ 1675us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 04:35:26.790561 (+ 1928us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.92.62:43939: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1713,"thread_start_us":93,"threads_started":1}
I20260809 04:35:26.791704 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 27628
E20260809 04:35:26.806432 27779 webserver.cc:533] Webserver: sslize failed:
E20260809 04:35:26.806537 27779 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-09T04:35:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T04:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:26Z Disabled control of system clock
I20260809 04:35:26.849689 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:44035
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:44035
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/testchainca.cert with env {}
W20260809 04:35:26.962348 27785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:26.962631 27785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:26.962671 27785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:26.966696 27785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:26.966753 27785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:26.966768 27785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:26.966782 27785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:26.966796 27785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 04:35:26.966817 27785 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 04:35:26.966833 27785 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 04:35:26.972136 27785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:44035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:44035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:26.973171 27785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:26.974215 27785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:26.982612 27791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:26.982646 27790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:26.982647 27795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:26.982831 27785 server_base.cc:1034] running on GCE node
I20260809 04:35:26.983455 27785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:26.984586 27785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:26.985756 27785 hybrid_clock.cc:648] HybridClock initialized: now 1786250126985735 us; error 34 us; skew 500 ppm
I20260809 04:35:26.987605 27785 webserver.cc:460] Webserver started at http://127.19.92.62:43505/ using document root <none> and password file <none>
I20260809 04:35:26.988171 27785 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:26.988234 27785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:26.988436 27785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:26.990649 27785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "6e4aad0b9e6a43ae948beb7567ef27ef"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "384faec170e8f7caeef211698e1b7156"
server_key_iv: "ae5bc5955de5314357ec68cf99772ffd"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.991226 27785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "6e4aad0b9e6a43ae948beb7567ef27ef"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "384faec170e8f7caeef211698e1b7156"
server_key_iv: "ae5bc5955de5314357ec68cf99772ffd"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.995198 27785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260809 04:35:26.997848 27805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:26.999106 27785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:35:26.999228 27785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "6e4aad0b9e6a43ae948beb7567ef27ef"
format_stamp: "Formatted at 2026-08-09 04:35:26 on dist-test-slave-rmhg"
server_key: "384faec170e8f7caeef211698e1b7156"
server_key_iv: "ae5bc5955de5314357ec68cf99772ffd"
server_key_version: "encryptionkey@0"
I20260809 04:35:26.999331 27785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:27.017828 27785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:27.018602 27785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:27.018795 27785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:27.025497 27785 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:44035
I20260809 04:35:27.025512 27902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:44035 every 8 connection(s)
I20260809 04:35:27.026456 27785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:27.029312 27903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:27.035220 27903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef: Bootstrap starting.
I20260809 04:35:27.035547 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 27785
I20260809 04:35:27.035648 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:27.035879 19824 external_mini_cluster.cc:1428] Setting key 126584eb5ac2dde0c4d83b43a4315b7c
I20260809 04:35:27.037889 27903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:27.038628 27903 log.cc:824] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:27.040668 27903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef: No bootstrap required, opened a new log
I20260809 04:35:27.044123 27903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aad0b9e6a43ae948beb7567ef27ef" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44035 } }
I20260809 04:35:27.044348 27903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:27.044409 27903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4aad0b9e6a43ae948beb7567ef27ef, State: Initialized, Role: FOLLOWER
I20260809 04:35:27.044865 27903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aad0b9e6a43ae948beb7567ef27ef" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44035 } }
I20260809 04:35:27.044996 27903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:27.045059 27903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:27.045182 27903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:27.045435 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:27.037295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.19.92.62:44035)
0809 04:35:27.038062 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:27.038078 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:27.038118 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:35:27.038791 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:27.038811 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:27.039302 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:27.039696 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:27.040180 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:27.041248 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:27.042102 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:27.042592 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:27.042755 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:27.043527 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:27.043532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:27.043546 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:35:27.043939 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:27.043995 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:35:27.044015 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:27.044102 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:27.044248 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 04:35:27.044550 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":132,"threads_started":1}
I20260809 04:35:27.046103 27903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aad0b9e6a43ae948beb7567ef27ef" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44035 } }
I20260809 04:35:27.046450 27903 leader_election.cc:302] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4aad0b9e6a43ae948beb7567ef27ef; no voters:
I20260809 04:35:27.046787 27903 leader_election.cc:288] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:27.046901 27910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:27.047116 27910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [term 1 LEADER]: Becoming Leader. State: Replica: 6e4aad0b9e6a43ae948beb7567ef27ef, State: Running, Role: LEADER
I20260809 04:35:27.047434 27910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aad0b9e6a43ae948beb7567ef27ef" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44035 } }
I20260809 04:35:27.048026 27903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:27.048426 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:35:27.049242 27912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e4aad0b9e6a43ae948beb7567ef27ef. Latest consensus state: current_term: 1 leader_uuid: "6e4aad0b9e6a43ae948beb7567ef27ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aad0b9e6a43ae948beb7567ef27ef" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44035 } } }
I20260809 04:35:27.049207 27911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e4aad0b9e6a43ae948beb7567ef27ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aad0b9e6a43ae948beb7567ef27ef" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44035 } } }
I20260809 04:35:27.049348 27912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:27.049348 27911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:27.050122 27925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:27.052919 27925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:27.058326 27925 catalog_manager.cc:1347] Generated new cluster ID: 02b2d302bbfb45eab6a0f5c7d05f11b6
I20260809 04:35:27.058394 27925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:27.073547 27925 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:27.074781 27925 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:27.080876 27925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e4aad0b9e6a43ae948beb7567ef27ef: Generated new TSK 0
I20260809 04:35:27.081774 27925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:35:27.166672 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:27.099499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53044 (local address 127.19.92.62:44035)
0809 04:35:27.099631 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:27.099636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:27.099879 (+ 243us) server_negotiation.cc:407] Connection header received
0809 04:35:27.100033 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:27.100036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:27.100088 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:27.100181 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0809 04:35:27.100498 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:27.101055 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:27.102333 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:27.102692 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:27.102789 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:27.103115 (+ 326us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 04:35:27.166273 (+ 63158us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 04:35:27.166552 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260809 04:35:27.166754 27906 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 04:35:27.166976 27949 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:27.099337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:44035 (local address 127.0.0.1:53044)
0809 04:35:27.099776 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:27.099791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:27.099909 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:27.100189 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:27.100192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:27.100200 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0809 04:35:27.100315 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:27.100322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:27.101195 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:27.101198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:27.102189 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:27.102200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:27.102817 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:27.102821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:27.102910 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:27.102942 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 04:35:27.166531 (+ 63589us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 04:35:27.166772 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.92.62:44035: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":123,"threads_started":1}
I20260809 04:35:27.167661 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 27785
2026-08-09T04:35:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (354 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:27 dist-test-slave-rmhg krb5kdc[27964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:27 dist-test-slave-rmhg krb5kdc[27964](info): set up 2 sockets
Aug 09 04:35:27 dist-test-slave-rmhg krb5kdc[27964](info): commencing operation
krb5kdc: starting...
W20260809 04:35:28.559728 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:28 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:28.687309 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:40897
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:28.800352 27986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:28.800657 27986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:28.800717 27986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:28.804816 27986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:28.804885 27986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:28.804914 27986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:28.804935 27986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:28.804955 27986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:28.810410 27986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:40897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:28.811497 27986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:28.812541 27986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:28.821017 27994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:28.821110 27986 server_base.cc:1034] running on GCE node
W20260809 04:35:28.821024 27995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:28.821012 27999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:28.821836 27986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:28.822925 27986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:28.824092 27986 hybrid_clock.cc:648] HybridClock initialized: now 1786250128824077 us; error 27 us; skew 500 ppm
Aug 09 04:35:28 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250128, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:28.826829 27986 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:28.827714 27986 webserver.cc:460] Webserver started at http://127.19.92.62:33247/ using document root <none> and password file <none>
I20260809 04:35:28.828243 27986 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:28.828311 27986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:28.828544 27986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:28.830612 27986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "541fcf9e6ec94082b6542ebaad90a6ca"
format_stamp: "Formatted at 2026-08-09 04:35:28 on dist-test-slave-rmhg"
server_key: "d64287ccde925dc4ee269087f2cc747b"
server_key_iv: "fce917482c06fc2d310846f17edf3ae5"
server_key_version: "encryptionkey@0"
I20260809 04:35:28.831151 27986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "541fcf9e6ec94082b6542ebaad90a6ca"
format_stamp: "Formatted at 2026-08-09 04:35:28 on dist-test-slave-rmhg"
server_key: "d64287ccde925dc4ee269087f2cc747b"
server_key_iv: "fce917482c06fc2d310846f17edf3ae5"
server_key_version: "encryptionkey@0"
I20260809 04:35:28.835031 27986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:35:28.837636 28012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:28.838891 27986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:28.839026 27986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "541fcf9e6ec94082b6542ebaad90a6ca"
format_stamp: "Formatted at 2026-08-09 04:35:28 on dist-test-slave-rmhg"
server_key: "d64287ccde925dc4ee269087f2cc747b"
server_key_iv: "fce917482c06fc2d310846f17edf3ae5"
server_key_version: "encryptionkey@0"
I20260809 04:35:28.839134 27986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:28.854760 27986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:28.857812 27986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:28.857995 27986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:28.864949 27986 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:40897
I20260809 04:35:28.865036 28098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:40897 every 8 connection(s)
I20260809 04:35:28.865952 27986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:28.868700 28100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:28.874050 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 27986
I20260809 04:35:28.874191 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:28.874437 19824 external_mini_cluster.cc:1428] Setting key fc68ade6f4b877eec40cbaadd8e65e51
I20260809 04:35:28.874473 28100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca: Bootstrap starting.
I20260809 04:35:28.877301 28100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:28.878242 28100 log.cc:824] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:28.880546 28100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca: No bootstrap required, opened a new log
Aug 09 04:35:28 dist-test-slave-rmhg krb5kdc[27964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:28.884841 28100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541fcf9e6ec94082b6542ebaad90a6ca" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40897 } }
I20260809 04:35:28.885185 28100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:28.885251 28100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541fcf9e6ec94082b6542ebaad90a6ca, State: Initialized, Role: FOLLOWER
I20260809 04:35:28.885782 28100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541fcf9e6ec94082b6542ebaad90a6ca" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40897 } }
I20260809 04:35:28.885955 28100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:28.886030 28100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:28.886137 28100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:28.887245 28100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541fcf9e6ec94082b6542ebaad90a6ca" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40897 } }
I20260809 04:35:28.887626 28100 leader_election.cc:302] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 541fcf9e6ec94082b6542ebaad90a6ca; no voters:
I20260809 04:35:28.887988 28100 leader_election.cc:288] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:28.888083 28105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:28.888293 28105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [term 1 LEADER]: Becoming Leader. State: Replica: 541fcf9e6ec94082b6542ebaad90a6ca, State: Running, Role: LEADER
I20260809 04:35:28.888607 28105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541fcf9e6ec94082b6542ebaad90a6ca" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40897 } }
I20260809 04:35:28.889393 28100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:28.890420 28107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541fcf9e6ec94082b6542ebaad90a6ca. Latest consensus state: current_term: 1 leader_uuid: "541fcf9e6ec94082b6542ebaad90a6ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541fcf9e6ec94082b6542ebaad90a6ca" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40897 } } }
I20260809 04:35:28.890547 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:28.875935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.19.92.62:40897)
0809 04:35:28.876397 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:28.876416 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:28.876469 (+ 53us) server_negotiation.cc:407] Connection header received
0809 04:35:28.877346 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:28.877374 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:28.877845 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:28.878229 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:28.878979 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:28.880466 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:28.881376 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:28.882079 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:28.882240 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:28.884797 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:28.884802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:28.884819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:28.884851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:28.886868 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:28.887293 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:28.887299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:28.887305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:28.887460 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:28.887710 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:28.887713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:28.887715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:28.888249 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:28.889034 (+ 785us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:28.889211 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 04:35:28.889529 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":127,"threads_started":1}
I20260809 04:35:28.890595 28106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "541fcf9e6ec94082b6542ebaad90a6ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541fcf9e6ec94082b6542ebaad90a6ca" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40897 } } }
I20260809 04:35:28.890671 28106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:28.890554 28107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:28.891124 28119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:28.894359 28119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:28.899782 28119 catalog_manager.cc:1347] Generated new cluster ID: 3562db439db04ddf9b12812a8d430e9f
I20260809 04:35:28.899858 28119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:28.910447 28119 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:28.911769 28119 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:28.918690 28119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 541fcf9e6ec94082b6542ebaad90a6ca: Generated new TSK 0
I20260809 04:35:28.919618 28119 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:28.954092 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:40897
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:29.065690 28140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:29.065940 28140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:29.066006 28140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:29.069892 28140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:29.069947 28140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:29.070029 28140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:29.075141 28140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:29.076285 28140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:29.077358 28140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:29.087000 28151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:29.086993 28146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:29.087111 28140 server_base.cc:1034] running on GCE node
W20260809 04:35:29.086993 28147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:29.087661 28140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:29.088272 28140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:29.089421 28140 hybrid_clock.cc:648] HybridClock initialized: now 1786250129089396 us; error 41 us; skew 500 ppm
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.092180 28140 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:29.093042 28140 webserver.cc:460] Webserver started at http://127.19.92.1:37635/ using document root <none> and password file <none>
I20260809 04:35:29.093622 28140 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:29.093685 28140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:29.093858 28140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:29.095835 28140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "0351553345ed456884b6c3c15dc7685c"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "3f417afaf826cb65d9b39477ff658dac"
server_key_iv: "1415be27f6632e6073ed81908d57f43f"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.096352 28140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "0351553345ed456884b6c3c15dc7685c"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "3f417afaf826cb65d9b39477ff658dac"
server_key_iv: "1415be27f6632e6073ed81908d57f43f"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.100041 28140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 04:35:29.102506 28163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.103708 28140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:29.103811 28140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "0351553345ed456884b6c3c15dc7685c"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "3f417afaf826cb65d9b39477ff658dac"
server_key_iv: "1415be27f6632e6073ed81908d57f43f"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.103917 28140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:29.122836 28140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:29.125937 28140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:29.126116 28140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:29.126724 28140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:29.127573 28140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:29.127622 28140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.127743 28140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:29.127812 28140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.136170 28140 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:43851
I20260809 04:35:29.136198 28331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:43851 every 8 connection(s)
I20260809 04:35:29.137063 28140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:29.141846 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 28140
I20260809 04:35:29.141970 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:29.142297 19824 external_mini_cluster.cc:1428] Setting key 156b50d0d20ce14ff399be5dd54fa786
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:29.151345 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.138856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:35245 (local address 127.19.92.62:40897)
0809 04:35:29.139031 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.139036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.139863 (+ 827us) server_negotiation.cc:407] Connection header received
0809 04:35:29.141168 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.141171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.141230 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.141308 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.142645 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.143199 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.144707 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.144939 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.145020 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.147457 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:29.147461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:29.147464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:29.147517 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:29.149322 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.149816 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.149820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.149821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.149918 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.150217 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.150221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.150222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.150465 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:29.150577 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:29.151034 (+ 457us) server_negotiation.cc:299] Negotiation successful
0809 04:35:29.151171 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:35:29.152139 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.139266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:40897 (local address 127.19.92.1:35245)
0809 04:35:29.139714 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:29.139757 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:29.140922 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:29.141463 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:29.141470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:29.141887 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:29.142502 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:29.142518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.143360 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.143363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:29.144593 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:29.144605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.145079 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.145083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:29.145308 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.145913 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:29.145938 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:29.147290 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:29.149434 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:29.149439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:29.149452 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:29.149719 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:29.150002 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:29.150004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:29.150005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:29.150135 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:29.150563 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:29.150568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:29.150809 (+ 241us) client_negotiation.cc:769] Sending connection context
0809 04:35:29.150986 (+ 177us) client_negotiation.cc:240] Negotiation successful
0809 04:35:29.151247 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:35:29.153415 28332 heartbeater.cc:347] Connected to a master server at 127.19.92.62:40897
I20260809 04:35:29.153705 28332 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:29.154186 28332 heartbeater.cc:511] Master 127.19.92.62:40897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:35:29.155956 28038 ts_manager.cc:194] Registered new tserver with Master: 0351553345ed456884b6c3c15dc7685c (127.19.92.1:43851)
I20260809 04:35:29.157509 28038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:35245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:29.171481 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:40897
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:29.284940 28338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:29.285257 28338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:29.285307 28338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:29.289317 28338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:29.289371 28338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:29.289428 28338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:29.294778 28338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:29.295823 28338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:29.296875 28338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:29.306438 28347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:29.306411 28345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:29.306413 28344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:29.306505 28338 server_base.cc:1034] running on GCE node
I20260809 04:35:29.307019 28338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:29.307715 28338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:29.308990 28338 hybrid_clock.cc:648] HybridClock initialized: now 1786250129308968 us; error 37 us; skew 500 ppm
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.315929 28338 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:29.317065 28338 webserver.cc:460] Webserver started at http://127.19.92.2:42311/ using document root <none> and password file <none>
I20260809 04:35:29.317704 28338 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:29.317780 28338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:29.318009 28338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:29.320231 28338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "2220aaf288754d45ad34f5643fe78168"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "0267d502d0f8659d06ac7032d1c11f46"
server_key_iv: "e3934286a0cbe1c1524fa48bfb9a86be"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.320797 28338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "2220aaf288754d45ad34f5643fe78168"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "0267d502d0f8659d06ac7032d1c11f46"
server_key_iv: "e3934286a0cbe1c1524fa48bfb9a86be"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.324785 28338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 04:35:29.327816 28355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.329221 28338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 04:35:29.329344 28338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "2220aaf288754d45ad34f5643fe78168"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "0267d502d0f8659d06ac7032d1c11f46"
server_key_iv: "e3934286a0cbe1c1524fa48bfb9a86be"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.329455 28338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:29.346159 28338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:29.349881 28338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:29.350087 28338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:29.350762 28338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:29.351609 28338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:29.351658 28338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.351712 28338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:29.351742 28338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.360141 28338 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:36393
I20260809 04:35:29.360190 28540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:36393 every 8 connection(s)
I20260809 04:35:29.361070 28338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:29.368815 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 28338
I20260809 04:35:29.368933 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:29.369272 19824 external_mini_cluster.cc:1428] Setting key 284dff28fad24fb72c865a18fbeb356c
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:29.375608 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.363103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:40823 (local address 127.19.92.62:40897)
0809 04:35:29.363251 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.363255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.364150 (+ 895us) server_negotiation.cc:407] Connection header received
0809 04:35:29.365576 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.365580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.365641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.365719 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.366977 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.367515 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.368681 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.368926 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.369024 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.372126 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:29.372129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:29.372131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:29.372157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:29.373581 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.374067 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.374077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.374079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.374172 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.374462 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.374465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.374466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.374685 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:29.374773 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:29.375319 (+ 546us) server_negotiation.cc:299] Negotiation successful
0809 04:35:29.375450 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:35:29.376478 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.363475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:40897 (local address 127.19.92.2:40823)
0809 04:35:29.363988 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:29.364033 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:29.365348 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:29.365921 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:29.365928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:29.366300 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:29.366821 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:29.366835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.367639 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.367642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:29.368551 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:29.368561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.369216 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.369221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:29.369425 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.370046 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:29.370064 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:29.371936 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:29.373689 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:29.373694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:29.373709 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:29.373963 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:29.374249 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:29.374251 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:29.374252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:29.374385 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:29.374774 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:29.374779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:29.375064 (+ 285us) client_negotiation.cc:769] Sending connection context
0809 04:35:29.375264 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 04:35:29.375548 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260809 04:35:29.377655 28541 heartbeater.cc:347] Connected to a master server at 127.19.92.62:40897
I20260809 04:35:29.377935 28541 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:29.378580 28541 heartbeater.cc:511] Master 127.19.92.62:40897 requested a full tablet report, sending...
I20260809 04:35:29.379792 28038 ts_manager.cc:194] Registered new tserver with Master: 2220aaf288754d45ad34f5643fe78168 (127.19.92.2:36393)
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:35:29.380357 28038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:40823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:29.399906 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:40897
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:29.514087 28551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:29.514374 28551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:29.514420 28551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:29.518503 28551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:29.518569 28551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:29.518628 28551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:29.523921 28551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:40897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:29.525002 28551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:29.526055 28551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:29.535598 28557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:29.535588 28558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:29.535712 28551 server_base.cc:1034] running on GCE node
W20260809 04:35:29.535588 28565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:29.536314 28551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:29.536919 28551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:29.538100 28551 hybrid_clock.cc:648] HybridClock initialized: now 1786250129538062 us; error 45 us; skew 500 ppm
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.540745 28551 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:29.541616 28551 webserver.cc:460] Webserver started at http://127.19.92.3:37155/ using document root <none> and password file <none>
I20260809 04:35:29.542155 28551 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:29.542210 28551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:29.542438 28551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:29.544531 28551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "41d90774e15744be90c3d61979396b3d"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "2cca6003ee9fe5e655b6e0ab76c7323a"
server_key_iv: "36f59ffb35f523653833e72e3d2d9adb"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.545063 28551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "41d90774e15744be90c3d61979396b3d"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "2cca6003ee9fe5e655b6e0ab76c7323a"
server_key_iv: "36f59ffb35f523653833e72e3d2d9adb"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.548640 28551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:29.551017 28575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.552256 28551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:29.552378 28551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "41d90774e15744be90c3d61979396b3d"
format_stamp: "Formatted at 2026-08-09 04:35:29 on dist-test-slave-rmhg"
server_key: "2cca6003ee9fe5e655b6e0ab76c7323a"
server_key_iv: "36f59ffb35f523653833e72e3d2d9adb"
server_key_version: "encryptionkey@0"
I20260809 04:35:29.552510 28551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:29.583725 28551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:29.587143 28551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:29.587342 28551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:29.587983 28551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:29.588886 28551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:29.588974 28551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.589043 28551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:29.589075 28551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:29.598302 28551 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:37365
I20260809 04:35:29.598331 28740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:37365 every 8 connection(s)
I20260809 04:35:29.599244 28551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:29.607554 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 28551
I20260809 04:35:29.607692 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:29.607996 19824 external_mini_cluster.cc:1428] Setting key 06e04a29c4b5cfcc7f9cca815ced1810
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:29.613476 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.601231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:33557 (local address 127.19.92.62:40897)
0809 04:35:29.601357 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.601361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.602269 (+ 908us) server_negotiation.cc:407] Connection header received
0809 04:35:29.603694 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.603698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.603762 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.603841 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.605220 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.605760 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.606989 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.607245 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.607356 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.610307 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:29.610310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:29.610312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:29.610342 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:29.611667 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.612141 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.612144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.612145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.612223 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.612485 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.612487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.612489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.612660 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:29.612729 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:29.613239 (+ 510us) server_negotiation.cc:299] Negotiation successful
0809 04:35:29.613353 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:35:29.614358 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.601602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:40897 (local address 127.19.92.3:33557)
0809 04:35:29.602105 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:29.602154 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:29.603434 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:29.604044 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:29.604052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:29.604458 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:29.605029 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:29.605047 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.605896 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.605899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:29.606841 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:29.606853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.607421 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.607424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:29.607707 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.608367 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:29.608385 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:29.610157 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:29.611783 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:29.611788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:29.611801 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:29.612071 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:29.612287 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:29.612289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:29.612290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:29.612418 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:29.612729 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:29.612734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:29.612982 (+ 248us) client_negotiation.cc:769] Sending connection context
0809 04:35:29.613138 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 04:35:29.613465 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260809 04:35:29.615532 28741 heartbeater.cc:347] Connected to a master server at 127.19.92.62:40897
I20260809 04:35:29.615767 28741 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:29.616259 28741 heartbeater.cc:511] Master 127.19.92.62:40897 requested a full tablet report, sending...
I20260809 04:35:29.617303 28038 ts_manager.cc:194] Registered new tserver with Master: 41d90774e15744be90c3d61979396b3d (127.19.92.3:37365)
I20260809 04:35:29.617841 28038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:33557
I20260809 04:35:29.621893 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/test-admin.keytab.
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.638552 19824 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 04:35:29.641595 28750 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:29.640034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.19.92.1:43851)
0809 04:35:29.640306 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.640324 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.640351 (+ 27us) server_negotiation.cc:407] Connection header received
0809 04:35:29.640408 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.640414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.640647 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.640783 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.641414 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40196: BlockingRecv error: recv got EOF from 127.0.0.1:40196 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
W20260809 04:35:29.641696 19824 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5cc' of type FILE
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.642827 19824 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.643800 19824 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:35:29.653234 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.644693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.19.92.1:43851)
0809 04:35:29.644812 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.644816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.644903 (+ 87us) server_negotiation.cc:407] Connection header received
0809 04:35:29.645000 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.645002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.645040 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.645101 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.645656 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.646487 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.647291 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.647708 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.647788 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.649961 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:29.649970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:29.649983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:29.650006 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:29.651526 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.651935 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.651940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.651941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.652013 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.652205 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.652207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.652208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.652434 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:29.652685 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:29.652899 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 04:35:29.653050 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260809 04:35:29.656941 28750 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:29.655950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.19.92.1:43851)
0809 04:35:29.656069 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.656077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.656113 (+ 36us) server_negotiation.cc:407] Connection header received
0809 04:35:29.656219 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.656221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.656254 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.656313 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.656853 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40212: BlockingRecv error: recv got EOF from 127.0.0.1:40212 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 04:35:29.657083 19824 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5cc' of type FILE
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.658051 19824 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.658850 19824 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:35:29.666843 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.659621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.19.92.1:43851)
0809 04:35:29.659734 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.659737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.659768 (+ 31us) server_negotiation.cc:407] Connection header received
0809 04:35:29.659866 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.659868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.659900 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.659958 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.660370 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.660889 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.661695 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.661909 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.661978 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.664017 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:29.664020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:29.664022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:29.664045 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:29.665383 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.665753 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.665755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.665756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.665825 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.666083 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.666085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.666086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.666235 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:29.666326 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:29.666602 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 04:35:29.666694 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 04:35:29.669440 28750 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:29.668549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.19.92.1:43851)
0809 04:35:29.668660 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.668664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.668732 (+ 68us) server_negotiation.cc:407] Connection header received
0809 04:35:29.668845 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.668848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.668885 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.668945 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.669346 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40236: BlockingRecv error: recv got EOF from 127.0.0.1:40236 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 04:35:29.669811 19824 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250097044341-19824-0/krb5kdc/krb5cc)
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.670771 19824 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:29.671639 19824 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[27964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:35:29.679811 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:29.672384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.19.92.1:43851)
0809 04:35:29.672511 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:29.672514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:29.672560 (+ 46us) server_negotiation.cc:407] Connection header received
0809 04:35:29.672660 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:29.672663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:29.672696 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:29.672749 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:29.673224 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.673721 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.674538 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:29.674748 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:29.674820 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:29.677109 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:29.677114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:29.677116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:29.677141 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:29.678335 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.678703 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.678705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.678706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.678774 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:29.679075 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:29.679078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:29.679079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:29.679255 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:29.679354 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:29.679558 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 04:35:29.679650 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:35:29.680845 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 28140
I20260809 04:35:29.690711 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 28338
I20260809 04:35:29.703626 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 28551
I20260809 04:35:29.730409 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 27986
2026-08-09T04:35:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2309 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[28769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[28769](info): set up 2 sockets
Aug 09 04:35:29 dist-test-slave-rmhg krb5kdc[28769](info): commencing operation
krb5kdc: starting...
W20260809 04:35:30.857911 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:30 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.62.keytab.
Entry for principal oryx/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.62.keytab.
I20260809 04:35:30.981643 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:34937
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.62.keytab
--principal=oryx/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:34937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:31.093817 28788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:31.094097 28788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:31.094136 28788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:31.098061 28788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:31.098114 28788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:31.098132 28788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:31.098147 28788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:31.098160 28788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:31.103371 28788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:34937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.62.keytab
--principal=oryx/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:31.104387 28788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:31.105404 28788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:31.113487 28799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:31.113534 28797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:31.113547 28801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:31.113639 28788 server_base.cc:1034] running on GCE node
I20260809 04:35:31.114424 28788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:31.115654 28788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:31.116850 28788 hybrid_clock.cc:648] HybridClock initialized: now 1786250131116819 us; error 48 us; skew 500 ppm
Aug 09 04:35:31 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:31.119812 28788 init.cc:377] Logged in from keytab as oryx/127.19.92.62@KRBTEST.COM (short username oryx)
I20260809 04:35:31.120823 28788 webserver.cc:460] Webserver started at http://127.19.92.62:46765/ using document root <none> and password file <none>
I20260809 04:35:31.121457 28788 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:31.121526 28788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:31.121711 28788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:31.123894 28788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "ce7fd35486204c12aef2ac2792ad5cec"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "5f42ea7884a31c569947a191137c8941"
server_key_iv: "cacc947cc38671c216b9bc224358a0bb"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.124466 28788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "ce7fd35486204c12aef2ac2792ad5cec"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "5f42ea7884a31c569947a191137c8941"
server_key_iv: "cacc947cc38671c216b9bc224358a0bb"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.128299 28788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:35:31.130760 28809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.132063 28788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:31.132179 28788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "ce7fd35486204c12aef2ac2792ad5cec"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "5f42ea7884a31c569947a191137c8941"
server_key_iv: "cacc947cc38671c216b9bc224358a0bb"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.132292 28788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:31.171836 28788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:31.174984 28788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:31.175168 28788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:31.182292 28788 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:34937
I20260809 04:35:31.182547 28888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:34937 every 8 connection(s)
I20260809 04:35:31.183322 28788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:31.186216 28890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:31.188373 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 28788
I20260809 04:35:31.188544 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:31.188841 19824 external_mini_cluster.cc:1428] Setting key 7568c052ae89367cb36d8bbb3956a36b
I20260809 04:35:31.192564 28890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: Bootstrap starting.
I20260809 04:35:31.194881 28890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:31.195648 28890 log.cc:824] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:35:31 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:31.197968 28890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: No bootstrap required, opened a new log
I20260809 04:35:31.201220 28890 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } }
I20260809 04:35:31.201460 28890 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:31.201522 28890 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7fd35486204c12aef2ac2792ad5cec, State: Initialized, Role: FOLLOWER
I20260809 04:35:31.201987 28890 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } }
I20260809 04:35:31.202127 28890 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:31.202194 28890 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:31.202296 28890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:31.203266 28890 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } }
I20260809 04:35:31.203604 28890 leader_election.cc:302] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce7fd35486204c12aef2ac2792ad5cec; no voters:
I20260809 04:35:31.203902 28890 leader_election.cc:288] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:31.204032 28898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:31.204121 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:31.190338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.19.92.62:34937)
0809 04:35:31.190792 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:31.190807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:31.190844 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:35:31.191675 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:31.191694 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:31.192037 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:31.192348 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:31.193213 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.194253 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.195147 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.195669 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.195826 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:31.198516 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:31.198522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:31.198536 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:31.198567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:31.200944 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:31.201450 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:31.201455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:31.201462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:31.201622 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:31.201908 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:31.201911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:31.201913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:31.202404 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:31.202613 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:31.202912 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 04:35:31.203223 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":133,"threads_started":1}
I20260809 04:35:31.204252 28898 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 1 LEADER]: Becoming Leader. State: Replica: ce7fd35486204c12aef2ac2792ad5cec, State: Running, Role: LEADER
I20260809 04:35:31.204627 28898 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } }
I20260809 04:35:31.205232 28890 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:31.207011 28900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7fd35486204c12aef2ac2792ad5cec. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } }
I20260809 04:35:31.207147 28900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:31.207501 28899 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } }
I20260809 04:35:31.207594 28899 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:31.207790 28915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:31.210453 28915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:31.215799 28915 catalog_manager.cc:1347] Generated new cluster ID: 1420b5688c154c7eb3506004a0575921
I20260809 04:35:31.215869 28915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:31.231278 28915 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:31.232614 28915 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:31.240443 28915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: Generated new TSK 0
I20260809 04:35:31.241407 28915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab.
Entry for principal oryx/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab.
I20260809 04:35:31.277076 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34937
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab
--principal=oryx/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:31.392313 28927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:31.392602 28927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:31.392668 28927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:31.396785 28927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:31.396878 28927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:31.396962 28927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:31.402545 28927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab
--principal=oryx/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:31.403682 28927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:31.404781 28927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:31.414305 28934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:31.414346 28932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:31.414369 28936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:31.414888 28927 server_base.cc:1034] running on GCE node
I20260809 04:35:31.415330 28927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:31.416018 28927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:31.417192 28927 hybrid_clock.cc:648] HybridClock initialized: now 1786250131417161 us; error 42 us; skew 500 ppm
Aug 09 04:35:31 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:31.419955 28927 init.cc:377] Logged in from keytab as oryx/127.19.92.1@KRBTEST.COM (short username oryx)
I20260809 04:35:31.420886 28927 webserver.cc:460] Webserver started at http://127.19.92.1:36641/ using document root <none> and password file <none>
I20260809 04:35:31.421520 28927 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:31.421589 28927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:31.421797 28927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:31.423894 28927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "2e4c93a7abd24a6492b3d1eb45c25149"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "873f5725554de9ef366d7c515c82cfce"
server_key_iv: "3093712298d34c0fcab01838e44e8e0a"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.424482 28927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "2e4c93a7abd24a6492b3d1eb45c25149"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "873f5725554de9ef366d7c515c82cfce"
server_key_iv: "3093712298d34c0fcab01838e44e8e0a"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.428292 28927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 04:35:31.430922 28944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.432201 28927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:31.432317 28927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "2e4c93a7abd24a6492b3d1eb45c25149"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "873f5725554de9ef366d7c515c82cfce"
server_key_iv: "3093712298d34c0fcab01838e44e8e0a"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.432417 28927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:31.445045 28927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:31.448311 28927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:31.448513 28927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:31.449111 28927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:31.450001 28927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:31.450057 28927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.450110 28927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:31.450137 28927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.458902 28927 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:45189
I20260809 04:35:31.458956 29107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:45189 every 8 connection(s)
I20260809 04:35:31.459864 28927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:31.464859 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 28927
I20260809 04:35:31.464993 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:31.465328 19824 external_mini_cluster.cc:1428] Setting key ad157d0f7f67c3c51c47567b76a8e5e4
Aug 09 04:35:31 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:31.473662 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:31.461782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:40471 (local address 127.19.92.62:34937)
0809 04:35:31.461927 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:31.461939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:31.462797 (+ 858us) server_negotiation.cc:407] Connection header received
0809 04:35:31.464109 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:31.464112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:31.464167 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:31.464241 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:31.465535 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.466166 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.467323 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.467563 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.467653 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:31.470314 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:31.470319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:31.470321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:31.470346 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:31.471647 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:31.472129 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:31.472133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:31.472134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:31.472210 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:31.472529 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:31.472532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:31.472534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:31.472720 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:31.472807 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:31.473360 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 04:35:31.473495 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:35:31.474529 29110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:31.462202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:40471)
0809 04:35:31.462654 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:31.462696 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:31.463911 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:31.464387 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:31.464394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:31.464806 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:31.465371 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:31.465387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.466296 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.466300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:31.467199 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:31.467210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.467708 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.467713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:31.467896 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:31.468489 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:31.468506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:31.470142 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:31.471757 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:31.471764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:31.471780 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:31.472035 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:31.472291 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:31.472294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:31.472296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:31.472441 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:31.472820 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:31.472827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:31.473084 (+ 257us) client_negotiation.cc:769] Sending connection context
0809 04:35:31.473292 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 04:35:31.473646 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:35:31.475907 29108 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34937
I20260809 04:35:31.476171 29108 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:31.476718 29108 heartbeater.cc:511] Master 127.19.92.62:34937 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.2@KRBTEST.COM" created.
I20260809 04:35:31.478511 28831 ts_manager.cc:194] Registered new tserver with Master: 2e4c93a7abd24a6492b3d1eb45c25149 (127.19.92.1:45189)
I20260809 04:35:31.479656 28831 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:40471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab.
Entry for principal oryx/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab.
I20260809 04:35:31.493206 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34937
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab
--principal=oryx/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:31.607373 29115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:31.607658 29115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:31.607702 29115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:31.611658 29115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:31.611709 29115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:31.611770 29115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:31.616909 29115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab
--principal=oryx/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:31.617959 29115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:31.618949 29115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:31.628384 29123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:31.628423 29115 server_base.cc:1034] running on GCE node
W20260809 04:35:31.628377 29122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:31.628392 29125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:31.629016 29115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:31.629730 29115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:31.630893 29115 hybrid_clock.cc:648] HybridClock initialized: now 1786250131630859 us; error 47 us; skew 500 ppm
Aug 09 04:35:31 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:31.633764 29115 init.cc:377] Logged in from keytab as oryx/127.19.92.2@KRBTEST.COM (short username oryx)
I20260809 04:35:31.634698 29115 webserver.cc:460] Webserver started at http://127.19.92.2:39083/ using document root <none> and password file <none>
I20260809 04:35:31.635288 29115 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:31.635362 29115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:31.635581 29115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:31.638125 29115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "404364fda6b846dcb4dedb42b23e7caf"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "38e5d42751a512e206ff2a09a73e4d9e"
server_key_iv: "2e921ffc2062a90d9ff007d66b4d07f6"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.638738 29115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "404364fda6b846dcb4dedb42b23e7caf"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "38e5d42751a512e206ff2a09a73e4d9e"
server_key_iv: "2e921ffc2062a90d9ff007d66b4d07f6"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.642809 29115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:35:31.645452 29134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.646823 29115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:35:31.646948 29115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "404364fda6b846dcb4dedb42b23e7caf"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "38e5d42751a512e206ff2a09a73e4d9e"
server_key_iv: "2e921ffc2062a90d9ff007d66b4d07f6"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.647056 29115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:31.656289 29115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:31.659365 29115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:31.659534 29115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:31.660125 29115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:31.660948 29115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:31.660995 29115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.661046 29115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:31.661075 29115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.669461 29115 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:45401
I20260809 04:35:31.669507 29291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:45401 every 8 connection(s)
I20260809 04:35:31.670380 29115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
Aug 09 04:35:31 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:31.680299 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 29115
I20260809 04:35:31.680416 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:31.680764 19824 external_mini_cluster.cc:1428] Setting key 12cffe0d7b8f38c82cd500238d1467b4
I20260809 04:35:31.684095 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:31.672213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:44419 (local address 127.19.92.62:34937)
0809 04:35:31.672335 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:31.672338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:31.673321 (+ 983us) server_negotiation.cc:407] Connection header received
0809 04:35:31.674635 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:31.674638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:31.674690 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:31.674767 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:31.675952 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.676497 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.677697 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.677932 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.678015 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:31.680492 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:31.680495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:31.680498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:31.680530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:31.682014 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:31.682602 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:31.682606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:31.682607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:31.682700 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:31.683009 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:31.683012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:31.683013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:31.683240 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:31.683330 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:31.683810 (+ 480us) server_negotiation.cc:299] Negotiation successful
0809 04:35:31.683939 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:35:31.684958 29294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:31.672667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:44419)
0809 04:35:31.673168 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:31.673212 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:31.674420 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:31.674911 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:31.674918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:31.675281 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:31.675821 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:31.675837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.676607 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.676610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:31.677580 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:31.677599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.678062 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.678065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:31.678212 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:31.678817 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:31.678834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:31.680293 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:31.682145 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:31.682151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:31.682168 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:31.682494 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:31.682786 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:31.682788 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:31.682789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:31.682931 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:31.683331 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:31.683336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:31.683581 (+ 245us) client_negotiation.cc:769] Sending connection context
0809 04:35:31.683742 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 04:35:31.684011 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260809 04:35:31.686132 29292 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34937
I20260809 04:35:31.686398 29292 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:31.686957 29292 heartbeater.cc:511] Master 127.19.92.62:34937 requested a full tablet report, sending...
I20260809 04:35:31.688112 28831 ts_manager.cc:194] Registered new tserver with Master: 404364fda6b846dcb4dedb42b23e7caf (127.19.92.2:45401)
I20260809 04:35:31.688683 28831 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:44419
WARNING: no policy specified for oryx/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab.
Entry for principal oryx/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab.
I20260809 04:35:31.708280 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34937
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab
--principal=oryx/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:31.824713 29297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:31.824961 29297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:31.825021 29297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:31.828965 29297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:31.829020 29297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:31.829099 29297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:31.834214 29297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab
--principal=oryx/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:31.835295 29297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:31.836361 29297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:31.846438 29308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:31.846417 29305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:31.846668 29297 server_base.cc:1034] running on GCE node
W20260809 04:35:31.846442 29306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:31.847281 29297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:31.847903 29297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:31.849066 29297 hybrid_clock.cc:648] HybridClock initialized: now 1786250131849037 us; error 42 us; skew 500 ppm
Aug 09 04:35:31 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:31.851869 29297 init.cc:377] Logged in from keytab as oryx/127.19.92.3@KRBTEST.COM (short username oryx)
I20260809 04:35:31.852767 29297 webserver.cc:460] Webserver started at http://127.19.92.3:36129/ using document root <none> and password file <none>
I20260809 04:35:31.853363 29297 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:31.853434 29297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:31.853669 29297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:31.855760 29297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "4e157049549f428d8798db9612b2b72c"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "fc8ad567a94428c44ac67de16108eb9b"
server_key_iv: "870e11181f1c88f16cd0243cfb72d448"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.856310 29297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "4e157049549f428d8798db9612b2b72c"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "fc8ad567a94428c44ac67de16108eb9b"
server_key_iv: "870e11181f1c88f16cd0243cfb72d448"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.860004 29297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 04:35:31.862334 29318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.863513 29297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:31.863629 29297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "4e157049549f428d8798db9612b2b72c"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "fc8ad567a94428c44ac67de16108eb9b"
server_key_iv: "870e11181f1c88f16cd0243cfb72d448"
server_key_version: "encryptionkey@0"
I20260809 04:35:31.863719 29297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:31.887864 29297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:31.891057 29297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:31.891235 29297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:31.891848 29297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:31.892670 29297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:31.892719 29297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.892771 29297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:31.892796 29297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:31.901346 29297 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:45393
I20260809 04:35:31.901383 29478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:45393 every 8 connection(s)
I20260809 04:35:31.902197 29297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:31.905390 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 29297
I20260809 04:35:31.905501 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:31.905822 19824 external_mini_cluster.cc:1428] Setting key d6a0ff4d836e02ee60ec57cb4b22c1b1
Aug 09 04:35:31 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:31.915129 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:31.903902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:48923 (local address 127.19.92.62:34937)
0809 04:35:31.904042 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:31.904045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:31.904846 (+ 801us) server_negotiation.cc:407] Connection header received
0809 04:35:31.906274 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:31.906278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:31.906336 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:31.906423 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:31.907664 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.908189 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.909368 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.909584 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.909671 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:31.912023 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:31.912026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:31.912028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:31.912052 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:31.913118 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:31.913636 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:31.913640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:31.913641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:31.913720 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:31.914031 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:31.914034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:31.914036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:31.914218 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:31.914302 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:31.914816 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 04:35:31.914958 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:35:31.916000 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:31.904233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:48923)
0809 04:35:31.904704 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:31.904746 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:31.906057 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:31.906622 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:31.906631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:31.906983 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:31.907537 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:31.907553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.908303 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.908306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:31.909259 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:31.909269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:31.909721 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:31.909726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:31.909901 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:31.910434 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:31.910477 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:31.911866 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:31.913263 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:31.913269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:31.913284 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:31.913542 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:31.913802 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:31.913805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:31.913807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:31.913945 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:31.914315 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:31.914322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:31.914587 (+ 265us) client_negotiation.cc:769] Sending connection context
0809 04:35:31.914759 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 04:35:31.915099 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":135,"threads_started":1}
I20260809 04:35:31.917217 29479 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34937
I20260809 04:35:31.917479 29479 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:31.917974 29479 heartbeater.cc:511] Master 127.19.92.62:34937 requested a full tablet report, sending...
I20260809 04:35:31.919095 28831 ts_manager.cc:194] Registered new tserver with Master: 4e157049549f428d8798db9612b2b72c (127.19.92.3:45393)
I20260809 04:35:31.919621 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:35:31.919751 28831 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:48923
WARNING: no policy specified for oryx/127.19.92.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.61.keytab.
Entry for principal oryx/127.19.92.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.61.keytab.
I20260809 04:35:31.947705 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.61:40731
--webserver_interface=127.19.92.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.61.keytab
--principal=oryx/127.19.92.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:34937,127.19.92.61:40731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:32.061641 29486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:32.061895 29486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:32.061937 29486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:32.065928 29486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:32.065984 29486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:32.066002 29486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:32.066018 29486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:32.066046 29486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:32.071343 29486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:34937,127.19.92.61:40731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.61.keytab
--principal=oryx/127.19.92.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.61:40731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.92.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
W20260809 04:35:32.071615 29486 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.92.62:34937,127.19.92.61:40731'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 04:35:32.072510 29486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:32.073581 29486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:32.082408 29494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.082408 29498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.082413 29496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.082803 29486 server_base.cc:1034] running on GCE node
I20260809 04:35:32.083379 29486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:32.084527 29486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:32.085690 29486 hybrid_clock.cc:648] HybridClock initialized: now 1786250132085655 us; error 49 us; skew 500 ppm
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:32.088449 29486 init.cc:377] Logged in from keytab as oryx/127.19.92.61@KRBTEST.COM (short username oryx)
I20260809 04:35:32.089442 29486 webserver.cc:460] Webserver started at http://127.19.92.61:44663/ using document root <none> and password file <none>
I20260809 04:35:32.089994 29486 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:32.090075 29486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:32.090266 29486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:32.092309 29486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data/instance:
uuid: "e1c9cc2ed7e8400186869797c90d58a9"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "17fa57432e8229892f1f902219110bbb"
server_key_iv: "d15d0f2c8f50eed5de8fc005c688e0a5"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.092861 29486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal/instance:
uuid: "e1c9cc2ed7e8400186869797c90d58a9"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "17fa57432e8229892f1f902219110bbb"
server_key_iv: "d15d0f2c8f50eed5de8fc005c688e0a5"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.096493 29486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:35:32.098840 29509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.100049 29486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:35:32.100181 29486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
uuid: "e1c9cc2ed7e8400186869797c90d58a9"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "17fa57432e8229892f1f902219110bbb"
server_key_iv: "d15d0f2c8f50eed5de8fc005c688e0a5"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.100276 29486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:32.120759 29486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:32.124009 29486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:32.124190 29486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:32.131294 29486 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.61:40731
I20260809 04:35:32.131553 29587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.61:40731 every 8 connection(s)
I20260809 04:35:32.132280 29486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
I20260809 04:35:32.134754 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 29486
I20260809 04:35:32.135288 29589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:32.140187 29589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.92.61@KRBTEST.COM
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.61@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:32.150763 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.136478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.19.92.61:40731)
0809 04:35:32.136901 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.136920 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.136957 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:35:32.137690 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.137710 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.138053 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.138338 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.139226 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.140252 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.141411 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.141994 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.142152 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.145437 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.145444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.145455 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.145486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.147433 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.147989 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.147995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.148002 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.148131 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.148417 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.148420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.148421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.148957 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.149208 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.149432 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.149730 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":111,"threads_started":1}
I20260809 04:35:32.151677 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.142132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:51510)
0809 04:35:32.142547 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.142595 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.142713 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.143020 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.143025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.143423 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.143591 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.143600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.144711 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.144714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.145615 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.145625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.146189 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.146193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.146323 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.146700 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.146715 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.148303 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.150120 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.150129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.150134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.150343 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.150596 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.150598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.150599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.150681 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.151163 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.151168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.151268 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.151383 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.151542 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":163,"threads_started":1}
I20260809 04:35:32.152674 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.142177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.19.92.62:34937)
0809 04:35:32.142298 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.142302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.142679 (+ 377us) server_negotiation.cc:407] Connection header received
0809 04:35:32.142844 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.142847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.142899 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.142967 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.143744 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.144556 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.145733 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.146071 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.146155 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.148444 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.148447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.148450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.148478 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.150020 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.150419 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.150422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.150424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.150513 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.150765 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.150768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.150769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.151061 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.151158 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.151476 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.151592 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:35:32.153730 29589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.61@KRBTEST.COM for oryx/127.19.92.61@KRBTEST.COM
I20260809 04:35:32.161100 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.154169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.0.0.1:54844)
0809 04:35:32.154267 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.154279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.154341 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.154514 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.154516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.154647 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.154737 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.154741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.155758 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.155760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.156288 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.156300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.156726 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.156728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.156795 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.157231 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.157243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.158391 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.159808 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.159811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.159812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.159992 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.160270 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.160272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.160273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.160335 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.160659 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.160661 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.160739 (+ 78us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.160814 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.160919 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 04:35:32.161103 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.154267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.19.92.61:40731)
0809 04:35:32.154367 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.154370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.154380 (+ 10us) server_negotiation.cc:407] Connection header received
0809 04:35:32.154406 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.154408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.154438 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.154519 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.154819 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.155668 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.156394 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.156640 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.156718 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.158541 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.158544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.158546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.158568 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.159714 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.160105 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.160112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.160114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.160194 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.160395 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.160397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.160398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.160593 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.160673 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.160879 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.160979 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 04:35:32.164229 29589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Bootstrap starting.
I20260809 04:35:32.166661 29589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:32.167385 29589 log.cc:824] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:32.169286 29589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: No bootstrap required, opened a new log
I20260809 04:35:32.172276 29589 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } }
I20260809 04:35:32.172473 29589 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:32.172523 29589 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c9cc2ed7e8400186869797c90d58a9, State: Initialized, Role: FOLLOWER
I20260809 04:35:32.172899 29589 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } }
I20260809 04:35:32.173342 29600 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } } }
I20260809 04:35:32.173461 29600 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:35:32.174044 29589 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 04:35:32.178682 29612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:35:32.178767 29612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:35:32.181017 29486 master_runner.cc:428] Detected that this master e1c9cc2ed7e8400186869797c90d58a9 is joining an existing cluster
I20260809 04:35:32.181068 29486 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.92.61:40731
I20260809 04:35:32.192085 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.184094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.19.92.61:40731)
0809 04:35:32.184195 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.184199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.184216 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:35:32.184251 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.184254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.184296 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.184377 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.184882 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.185681 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.186815 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.187147 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.187220 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.188508 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.188513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.188516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.188553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.190386 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.191060 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.191063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.191065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.191139 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.191342 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.191345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.191347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.191522 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.191579 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.191828 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.191928 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":39}
I20260809 04:35:32.192750 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.183820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.0.0.1:54858)
0809 04:35:32.184091 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.184104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.184183 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.184391 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.184393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.184645 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.184779 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.184786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.185804 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.185807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.186710 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.186719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.187324 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.187327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.187404 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.187948 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.187961 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.188361 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.190489 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.190501 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.190504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.190964 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.191215 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.191218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.191220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.191279 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.191625 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.191628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.191705 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.192429 (+ 724us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.192589 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":107,"threads_started":1}
I20260809 04:35:32.193481 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.183580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.19.92.62:34937)
0809 04:35:32.184584 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.184588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.184601 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:32.184638 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.184641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.184684 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.184757 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.185411 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.186207 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.187451 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.187766 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.187843 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.189850 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.189854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.189857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.189885 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.191424 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.191902 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.191905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.191907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.191990 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.192334 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.192337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.192339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.192555 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.192639 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.193220 (+ 581us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.193335 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912}
I20260809 04:35:32.193842 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.183175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:51512)
0809 04:35:32.183472 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.183487 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.183584 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.184877 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.184880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.185102 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.185271 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.185279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.186340 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.186343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.187311 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.187323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.188006 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.188009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.188114 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.188825 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.188840 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.189700 (+ 860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.191559 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.191562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.191565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.191795 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.192084 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.192087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.192089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.192239 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.192685 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.192688 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.192775 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.192867 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.193669 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.62@KRBTEST.COM for oryx/127.19.92.61@KRBTEST.COM
I20260809 04:35:32.208388 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.198633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.0.0.1:54874)
0809 04:35:32.199033 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.199076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.199186 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.199762 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.199769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.200162 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.200275 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.200281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.201017 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.201020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.201707 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.201718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.202132 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.202136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.202262 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.202751 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.202763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.204522 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.206491 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.206503 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.206511 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.206922 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.207206 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.207210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.207212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.207298 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.207732 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.207739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.207859 (+ 120us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.207984 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.208207 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":115,"threads_started":1}
I20260809 04:35:32.208451 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.199381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.19.92.61:40731)
0809 04:35:32.199515 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.199518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.199529 (+ 11us) server_negotiation.cc:407] Connection header received
0809 04:35:32.199588 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.199590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.199632 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.199698 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.200391 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.200898 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.201818 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.202030 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.202117 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.204723 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.204727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.204730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.204756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.206383 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.207036 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.207040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.207042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.207121 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.207379 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.207383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.207385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.207624 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.207713 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.208053 (+ 340us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.208322 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 04:35:32.209864 28831 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.92.61:40731: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9"
member_type: NON_VOTER
last_known_addr {
host: "127.19.92.61"
port: 40731
}
attrs {
promote: true
}
}
dest_uuid: "ce7fd35486204c12aef2ac2792ad5cec"
cas_config_opid_index: -1
I20260809 04:35:32.210487 28831 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } }
I20260809 04:35:32.212100 29627 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e1c9cc2ed7e8400186869797c90d58a9 (127.19.92.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } }
I20260809 04:35:32.212990 29551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:32.213511 29627 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.92.61:40731
I20260809 04:35:32.213904 29486 master.cc:537] Master@127.19.92.61:40731 shutting down...
I20260809 04:35:32.213891 29626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } }
I20260809 04:35:32.213986 29626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:32.214494 29551 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Refusing update from remote peer ce7fd35486204c12aef2ac2792ad5cec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 04:35:32.215253 29627 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.19.92.60@KRBTEST.COM; defaulting to no policy
W20260809 04:35:32.217801 28813 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.60@KRBTEST.COM" created.
W20260809 04:35:32.218775 28813 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec -> Peer e1c9cc2ed7e8400186869797c90d58a9 (127.19.92.61:40731): Couldn't send request to peer e1c9cc2ed7e8400186869797c90d58a9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 04:35:32.218796 29486 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:32.219067 29486 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:32.219175 29486 tablet_replica.cc:330] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: stopping tablet replica
I20260809 04:35:32.234267 29486 master.cc:552] Master@127.19.92.61:40731 shutdown complete.
I20260809 04:35:32.234338 29486 master_runner.cc:305] Clearing existing system tablet
I20260809 04:35:32.234612 29486 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:32.234676 29486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:32.236313 29486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 04:35:32.237385 29637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.237735 29486 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:32.237874 29486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:32.237928 29486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
uuid: "e1c9cc2ed7e8400186869797c90d58a9"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "17fa57432e8229892f1f902219110bbb"
server_key_iv: "d15d0f2c8f50eed5de8fc005c688e0a5"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.60.keytab.
Entry for principal oryx/127.19.92.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.60.keytab.
I20260809 04:35:32.240320 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.60:33155
--webserver_interface=127.19.92.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.60.keytab
--principal=oryx/127.19.92.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:34937,127.19.92.61:40731,127.19.92.60:33155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:35:32.241039 29486 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:32.243312 29486 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 04:35:32.243418 29486 log.cc:1196] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 04:35:32.243693 29486 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Deleting consensus metadata
I20260809 04:35:32.243863 29486 master_runner.cc:315] Copying system tablet from 127.19.92.62:34937
I20260809 04:35:32.245033 29486 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: tablet copy: Beginning tablet copy session from remote peer at address 127.19.92.62:34937
I20260809 04:35:32.254702 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.245753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.19.92.62:34937)
0809 04:35:32.245884 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.245888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.248090 (+ 2202us) server_negotiation.cc:407] Connection header received
0809 04:35:32.248220 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.248224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.248280 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.248348 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.248851 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.249452 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.250370 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.250609 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.250697 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.251865 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.251870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.251873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.251900 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.253204 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.253595 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.253599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.253601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.253678 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.253920 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.253924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.253925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.254102 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.254186 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.254437 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.254553 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 04:35:32.254712 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.245656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:51522)
0809 04:35:32.247960 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.247975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.248119 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.248381 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.248384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.248593 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.248720 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.248727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.249563 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.249567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.250245 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.250256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.250726 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.250730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.250826 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.251371 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.251383 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.251735 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.253309 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.253313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.253315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.253508 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.253758 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.253762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.253764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.253843 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.254189 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.254193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.254285 (+ 92us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.254370 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.254505 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2218,"thread_start_us":89,"threads_started":1}
I20260809 04:35:32.255191 28865 tablet_copy_service.cc:140] P ce7fd35486204c12aef2ac2792ad5cec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e1c9cc2ed7e8400186869797c90d58a9 ({username='oryx', principal='oryx/127.19.92.61@KRBTEST.COM'} at 127.0.0.1:51522)
I20260809 04:35:32.255358 28865 tablet_copy_service.cc:161] P ce7fd35486204c12aef2ac2792ad5cec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e1c9cc2ed7e8400186869797c90d58a9 at {username='oryx', principal='oryx/127.19.92.61@KRBTEST.COM'} at 127.0.0.1:51522: session id = e1c9cc2ed7e8400186869797c90d58a9-00000000000000000000000000000000
I20260809 04:35:32.256811 28865 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 04:35:32.258293 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 04:35:32.260177 29486 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: tablet copy: Starting download of 0 data blocks...
I20260809 04:35:32.260361 29486 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: tablet copy: Starting download of 1 WAL segments...
I20260809 04:35:32.261914 29486 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 04:35:32.263531 28865 tablet_copy_service.cc:342] P ce7fd35486204c12aef2ac2792ad5cec: Request end of tablet copy session e1c9cc2ed7e8400186869797c90d58a9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.92.61@KRBTEST.COM'} at 127.0.0.1:51522
I20260809 04:35:32.263645 28865 tablet_copy_service.cc:433] P ce7fd35486204c12aef2ac2792ad5cec: ending tablet copy session e1c9cc2ed7e8400186869797c90d58a9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e1c9cc2ed7e8400186869797c90d58a9
I20260809 04:35:32.270669 29486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:32.272318 29653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.272363 29651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.272580 29486 server_base.cc:1034] running on GCE node
W20260809 04:35:32.272673 29656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.272856 29486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:32.273144 29486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:32.274299 29486 hybrid_clock.cc:648] HybridClock initialized: now 1786250132274291 us; error 24 us; skew 500 ppm
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:32.275647 29486 init.cc:377] Logged in from keytab as oryx/127.19.92.61@KRBTEST.COM (short username oryx)
I20260809 04:35:32.276091 29486 webserver.cc:460] Webserver started at http://127.19.92.61:36739/ using document root <none> and password file <none>
I20260809 04:35:32.276283 29486 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:32.276319 29486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:32.281008 29486 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260809 04:35:32.282021 29668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.282241 29486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:32.282315 29486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
uuid: "e1c9cc2ed7e8400186869797c90d58a9"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "17fa57432e8229892f1f902219110bbb"
server_key_iv: "d15d0f2c8f50eed5de8fc005c688e0a5"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.282402 29486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:32.299115 29486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:32.300863 29486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:32.305827 29486 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.61:40731
I20260809 04:35:32.305856 29744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.61:40731 every 8 connection(s)
I20260809 04:35:32.306486 29486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
I20260809 04:35:32.308166 29746 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:35:32.308970 29746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Bootstrap starting.
I20260809 04:35:32.318562 29746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:35:32.318897 29746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: Bootstrap complete.
I20260809 04:35:32.319456 29746 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } }
I20260809 04:35:32.319591 29746 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1c9cc2ed7e8400186869797c90d58a9, State: Initialized, Role: LEARNER
I20260809 04:35:32.319698 29746 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } }
I20260809 04:35:32.320061 29753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } }
I20260809 04:35:32.320194 29746 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:32.320228 29753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: This master's current role is: LEARNER
I20260809 04:35:32.323136 29767 catalog_manager.cc:1259] Loaded cluster ID: 1420b5688c154c7eb3506004a0575921
I20260809 04:35:32.323256 29767 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: loading cluster ID for follower catalog manager: success
I20260809 04:35:32.324887 29767 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: acquiring CA information for follower catalog manager: success
I20260809 04:35:32.326074 29767 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 04:35:32.397702 29638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:32.398018 29638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:32.398062 29638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:32.402182 29638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:32.402254 29638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:32.402273 29638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:32.402289 29638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:32.402303 29638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:32.407869 29638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:34937,127.19.92.61:40731,127.19.92.60:33155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.60.keytab
--principal=oryx/127.19.92.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.60:33155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.92.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:32.408969 29638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:32.410030 29638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:32.418358 29774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.418491 29638 server_base.cc:1034] running on GCE node
W20260809 04:35:32.418385 29775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.418358 29777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.419219 29638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:32.420428 29638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:32.421594 29638 hybrid_clock.cc:648] HybridClock initialized: now 1786250132421566 us; error 39 us; skew 500 ppm
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:32.424163 29638 init.cc:377] Logged in from keytab as oryx/127.19.92.60@KRBTEST.COM (short username oryx)
I20260809 04:35:32.425057 29638 webserver.cc:460] Webserver started at http://127.19.92.60:44975/ using document root <none> and password file <none>
I20260809 04:35:32.425719 29638 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:32.425794 29638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:32.425998 29638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:32.427997 29638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data/instance:
uuid: "d649133147ca40869e119ca7e2fe6710"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "5d620efe3f261c51a3cc30848cb292be"
server_key_iv: "7c140826eb18fce86d2369cb68081ea7"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.428550 29638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal/instance:
uuid: "d649133147ca40869e119ca7e2fe6710"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "5d620efe3f261c51a3cc30848cb292be"
server_key_iv: "7c140826eb18fce86d2369cb68081ea7"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.432230 29638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:35:32.434801 29787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.436183 29638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:32.436319 29638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
uuid: "d649133147ca40869e119ca7e2fe6710"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "5d620efe3f261c51a3cc30848cb292be"
server_key_iv: "7c140826eb18fce86d2369cb68081ea7"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.436434 29638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:32.451942 29638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:32.454885 29638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:32.455073 29638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:32.462939 29638 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.60:33155
I20260809 04:35:32.463212 29856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.60:33155 every 8 connection(s)
I20260809 04:35:32.463994 29638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
I20260809 04:35:32.466784 29857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:32.469810 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 29638
I20260809 04:35:32.470813 29857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.60@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.92.60@KRBTEST.COM
I20260809 04:35:32.482169 29108 heartbeater.cc:503] Master 127.19.92.62:34937 was elected leader, sending a full tablet report...
I20260809 04:35:32.484548 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.472791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.19.92.62:34937)
0809 04:35:32.472965 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.472969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.473224 (+ 255us) server_negotiation.cc:407] Connection header received
0809 04:35:32.473392 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.473395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.473450 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.473527 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.474523 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.475339 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.476811 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.477206 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.477316 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.480445 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.480449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.480453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.480485 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.482026 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.482685 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.482689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.482690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.482787 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.483209 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.483213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.483215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.483482 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.483581 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.484233 (+ 652us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.484369 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 04:35:32.485898 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.471297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.19.92.60:33155)
0809 04:35:32.471875 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.471895 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.471937 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:35:32.472750 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.472774 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.473190 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.474247 (+ 1057us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.474347 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.475617 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.476799 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.477447 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.477628 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.480938 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.480946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.480961 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.480989 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.482758 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.483254 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.483260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.483261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.483342 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.483853 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.483855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.483856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.484107 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.484300 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.484551 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.485077 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":450,"server-negotiator.queue_time_us":424,"thread_start_us":163,"threads_started":1}
I20260809 04:35:32.485903 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.472726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:51528)
0809 04:35:32.473078 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.473128 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.473279 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.473584 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.473591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.473902 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.474379 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.474389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.475498 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.475502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.476695 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.476705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.477367 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.477371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.477602 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.478419 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.478441 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.480273 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.482145 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.482153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.482166 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.482592 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.482900 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.482904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.482905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.483093 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.483641 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.483649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.483989 (+ 340us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.484153 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.484994 (+ 841us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":42,"thread_start_us":112,"threads_started":1}
I20260809 04:35:32.487186 29857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.60@KRBTEST.COM for oryx/127.19.92.61@KRBTEST.COM
I20260809 04:35:32.494951 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.487665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.0.0.1:54884)
0809 04:35:32.487835 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.487848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.487918 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.488348 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.488350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.488520 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.488621 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.488627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.489534 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.489536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.490075 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.490084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.490473 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.490476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.490553 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.491004 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.491017 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.492267 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.493817 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.493819 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.493821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.494002 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.494205 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.494207 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.494208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.494274 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.494578 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.494580 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.494658 (+ 78us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.494728 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.494831 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260809 04:35:32.494992 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.487753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.19.92.61:40731)
0809 04:35:32.488147 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.488151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.488163 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:35:32.488200 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.488202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.488241 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.488316 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.488712 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.489428 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.490162 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.490381 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.490453 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.492407 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.492410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.492412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.492437 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.493722 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.494072 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.494074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.494075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.494147 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.494331 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.494333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.494334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.494495 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.494579 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.494772 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.494868 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":116,"threads_started":1}
I20260809 04:35:32.495733 29857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.60@KRBTEST.COM for oryx/127.19.92.60@KRBTEST.COM
I20260809 04:35:32.502283 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.496171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.0.0.1:37022)
0809 04:35:32.496281 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.496294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.496354 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.496520 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.496522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.496655 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.496726 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.496731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.497438 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.497440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.497940 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.497948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.498324 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.498327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.498386 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.498769 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.498780 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.499831 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.501115 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.501117 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.501119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.501332 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.501537 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.501538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.501540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.501634 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.501898 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.501900 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.501971 (+ 71us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.502040 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.502144 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260809 04:35:32.502314 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.496242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.19.92.60:33155)
0809 04:35:32.496343 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.496347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.496359 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:35:32.496408 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.496410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.496443 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.496503 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.496803 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.497344 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.498023 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.498255 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.498320 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.499947 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.499950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.499952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.499973 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.501012 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.501401 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.501404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.501405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.501474 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.501688 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.501690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.501691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.501837 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.501917 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.502081 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.502197 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":35}
I20260809 04:35:32.505721 29857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Bootstrap starting.
I20260809 04:35:32.508216 29857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:32.508942 29857 log.cc:824] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:32.511010 29857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: No bootstrap required, opened a new log
I20260809 04:35:32.514113 29857 raft_consensus.cc:348] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } }
I20260809 04:35:32.514319 29857 raft_consensus.cc:374] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:32.514377 29857 raft_consensus.cc:729] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d649133147ca40869e119ca7e2fe6710, State: Initialized, Role: FOLLOWER
I20260809 04:35:32.514811 29857 consensus_queue.cc:260] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } }
I20260809 04:35:32.515268 29863 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } } }
I20260809 04:35:32.515384 29863 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:35:32.516004 29857 sys_catalog.cc:564] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 04:35:32.521188 29877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:35:32.521271 29877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:35:32.524569 29638 master_runner.cc:428] Detected that this master d649133147ca40869e119ca7e2fe6710 is joining an existing cluster
I20260809 04:35:32.524634 29638 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.92.60:33155
I20260809 04:35:32.536022 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.527322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.19.92.61:40731)
0809 04:35:32.527543 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.527548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.528329 (+ 781us) server_negotiation.cc:407] Connection header received
0809 04:35:32.528476 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.528478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.528523 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.528570 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.529355 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.530112 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.531422 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.531756 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.531822 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.533033 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.533037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.533039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.533065 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.534422 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.534824 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.534827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.534830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.534909 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.535162 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.535165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.535167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.535339 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.535435 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.535720 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.535854 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260809 04:35:32.536062 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.527284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.0.0.1:54900)
0809 04:35:32.528000 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.528014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.528381 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.528689 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.528691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.529022 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.529239 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.529247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.530227 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.530230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.531294 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.531305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.531982 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.531986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.532079 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.532552 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.532564 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.532900 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.534524 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.534528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.534530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.534737 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.534990 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.534993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.534995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.535082 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.535443 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.535447 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.535570 (+ 123us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.535681 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.535869 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":127,"thread_start_us":105,"threads_started":1}
I20260809 04:35:32.539119 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 28927
I20260809 04:35:32.552222 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:45189
--local_ip_for_outbound_sockets=127.19.92.1
--tserver_master_addrs=127.19.92.62:34937
--webserver_port=36641
--webserver_interface=127.19.92.1
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab
--principal=oryx/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731,127.19.92.60:33155 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:35:32.554905 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.527485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.19.92.60:33155)
0809 04:35:32.527673 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.527677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.528285 (+ 608us) server_negotiation.cc:407] Connection header received
0809 04:35:32.528445 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.528448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.528484 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.528537 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.529294 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.530087 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.531691 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.531958 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.532022 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.550769 (+ 18747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.550773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.550776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.550801 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.552651 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.553506 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.553509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.553510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.553667 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.553983 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.553986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.553988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.554227 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.554312 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.554598 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.554729 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":124}
I20260809 04:35:32.555276 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.527640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.19.92.62:34937)
0809 04:35:32.527782 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.527787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.528340 (+ 553us) server_negotiation.cc:407] Connection header received
0809 04:35:32.528525 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.528529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.528593 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.528647 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.529457 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.530288 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.531499 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.531854 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.531927 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.550707 (+ 18780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.550712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.550715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.550751 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.553198 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.553857 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.553861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.553863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.553980 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.554289 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.554293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.554295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.554526 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.554600 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.554974 (+ 374us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.555112 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:35:32.555886 29887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.527777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:51532)
0809 04:35:32.528095 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.528109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.528416 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.528793 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.528796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.529083 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.529318 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.529327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.530430 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.530434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.531362 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.531374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.532081 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.532084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.532204 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.549783 (+ 17579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.549803 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.550442 (+ 639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.553340 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.553344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.553346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.553722 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.554096 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.554099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.554102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.554190 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.554674 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.554677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.554777 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.555482 (+ 705us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.555664 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":49,"thread_start_us":71,"threads_started":1}
I20260809 04:35:32.556284 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.527284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.0.0.1:37036)
0809 04:35:32.528062 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.528074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.528354 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.528669 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.528671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.529002 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.529174 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.529185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.530209 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.530212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.531591 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.531600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.532145 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.532148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.532229 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.550062 (+ 17833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.550079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.550603 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.553123 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.553126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.553128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.553407 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.553803 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.553805 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.553807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.553895 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.554354 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.554356 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.554452 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.555553 (+ 1101us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.555745 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":46,"thread_start_us":117,"threads_started":1}
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250131, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.62@KRBTEST.COM for oryx/127.19.92.60@KRBTEST.COM
I20260809 04:35:32.570070 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.560750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.19.92.60:33155)
0809 04:35:32.560883 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.560887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.560901 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:32.560980 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.560983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.561035 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.561112 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.561665 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.562446 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.563591 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.563943 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.564036 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.566634 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.566645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.566648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.566677 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.568332 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.568812 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.568815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.568818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.568905 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.569188 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.569191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.569193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.569423 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.569516 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.569785 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.569904 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:35:32.570070 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.560291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.0.0.1:37048)
0809 04:35:32.560482 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.560500 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.560592 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.561187 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.561191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.561402 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.561545 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.561554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.562566 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.562570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.563457 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.563471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.564060 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.564063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.564171 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.564704 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.564720 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.566418 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.568462 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.568465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.568468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.568722 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.568989 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.568991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.568993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.569086 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.569512 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.569515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.569626 (+ 111us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.569731 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.569886 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 04:35:32.571514 28831 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.92.60:33155: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d649133147ca40869e119ca7e2fe6710"
member_type: NON_VOTER
last_known_addr {
host: "127.19.92.60"
port: 33155
}
attrs {
promote: true
}
}
dest_uuid: "ce7fd35486204c12aef2ac2792ad5cec"
cas_config_opid_index: 5
I20260809 04:35:32.572217 28831 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } }
I20260809 04:35:32.573697 29626 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d649133147ca40869e119ca7e2fe6710 (127.19.92.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.575239 29626 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.92.60:33155
I20260809 04:35:32.575441 29626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.575541 29626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:32.575977 29638 master.cc:537] Master@127.19.92.60:33155 shutting down...
I20260809 04:35:32.576500 29824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:32.578326 29824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Refusing update from remote peer ce7fd35486204c12aef2ac2792ad5cec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 04:35:32.579068 29627 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 04:35:32.581622 28814 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec -> Peer d649133147ca40869e119ca7e2fe6710 (127.19.92.60:33155): Couldn't send request to peer d649133147ca40869e119ca7e2fe6710. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 04:35:32.582479 29638 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:32.582895 29638 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:32.583086 29638 tablet_replica.cc:330] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: stopping tablet replica
I20260809 04:35:32.585781 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.573659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.19.92.61:40731)
0809 04:35:32.573777 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.573781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.579899 (+ 6118us) server_negotiation.cc:407] Connection header received
0809 04:35:32.580712 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.580715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.580786 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.580852 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:32.581309 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.582101 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.583786 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.584906 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.584994 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.585179 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.585570 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.585652 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:35:32.586994 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.573545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.0.0.1:54912)
0809 04:35:32.579807 (+ 6262us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.579824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.580548 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.581032 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.581035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.581055 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:32.581177 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.581184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.582359 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.582363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.583641 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.583654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.585281 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.585284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.585380 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.585448 (+ 68us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.586635 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.586724 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6158}
I20260809 04:35:32.587544 29705 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 LEARNER]: Refusing update from remote peer ce7fd35486204c12aef2ac2792ad5cec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 04:35:32.587998 29627 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 04:35:32.588331 29638 master.cc:552] Master@127.19.92.60:33155 shutdown complete.
I20260809 04:35:32.588410 29638 master_runner.cc:305] Clearing existing system tablet
I20260809 04:35:32.588768 29638 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:32.588860 29638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:32.590008 29705 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d649133147ca40869e119ca7e2fe6710 (127.19.92.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.591954 29896 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.592139 29896 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: This master's current role is: LEARNER
I20260809 04:35:32.592373 29753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7fd35486204c12aef2ac2792ad5cec. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: NON_VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: true } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.592466 29753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: This master's current role is: LEARNER
I20260809 04:35:32.594257 29638 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260809 04:35:32.594753 29627 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: attempting to promote NON_VOTER e1c9cc2ed7e8400186869797c90d58a9 to VOTER
I20260809 04:35:32.595307 29627 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } }
I20260809 04:35:32.596909 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.596541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.0.0.1:37056)
0809 04:35:32.596732 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.596820 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260809 04:35:32.597179 29703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 LEARNER]: Refusing update from remote peer ce7fd35486204c12aef2ac2792ad5cec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 04:35:32.597793 29626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 04:35:32.597906 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.597441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.0.0.1:37072)
0809 04:35:32.597722 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.597773 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
W20260809 04:35:32.598002 28814 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec -> Peer d649133147ca40869e119ca7e2fe6710 (127.19.92.60:33155): Couldn't send request to peer d649133147ca40869e119ca7e2fe6710. Status: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33155: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 04:35:32.599341 29898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.599797 29627 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e1c9cc2ed7e8400186869797c90d58a9 (127.19.92.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.600353 29705 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e1c9cc2ed7e8400186869797c90d58a9 (127.19.92.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.601572 29627 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.601689 29627 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:32.601804 29627 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.601864 29627 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:32.602248 29753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.602360 29753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:35:32.602954 29638 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:32.603116 29638 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260809 04:35:32.603164 29638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
uuid: "d649133147ca40869e119ca7e2fe6710"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "5d620efe3f261c51a3cc30848cb292be"
server_key_iv: "7c140826eb18fce86d2369cb68081ea7"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.605547 29638 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:32.608095 29638 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 04:35:32.608215 29638 log.cc:1196] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 04:35:32.608484 29638 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Deleting consensus metadata
I20260809 04:35:32.608644 29638 master_runner.cc:315] Copying system tablet from 127.19.92.62:34937
I20260809 04:35:32.609966 29638 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: tablet copy: Beginning tablet copy session from remote peer at address 127.19.92.62:34937
I20260809 04:35:32.620075 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.610613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.19.92.62:34937)
0809 04:35:32.610736 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.610741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.610996 (+ 255us) server_negotiation.cc:407] Connection header received
0809 04:35:32.611115 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.611118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.611182 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.611266 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.612122 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.612946 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.614198 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.614561 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.614677 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.616221 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.616225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.616228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.616257 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.618135 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.618652 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.618656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.618658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.618748 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.619022 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.619025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.619027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.619348 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.619445 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.619733 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.619899 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:35:32.620071 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.610527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:51548)
0809 04:35:32.610873 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.610887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.611023 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.611293 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.611296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.611634 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.611976 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.611988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.613063 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.613067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.614059 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.614071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.614842 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.614846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.614963 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.615540 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.615556 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.616061 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.618275 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.618278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.618281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.618546 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.618837 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.618839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.618842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.618937 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.619441 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.619444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.619559 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.619665 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.619849 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":98,"threads_started":1}
I20260809 04:35:32.620797 28865 tablet_copy_service.cc:140] P ce7fd35486204c12aef2ac2792ad5cec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d649133147ca40869e119ca7e2fe6710 ({username='oryx', principal='oryx/127.19.92.60@KRBTEST.COM'} at 127.0.0.1:51548)
I20260809 04:35:32.620908 28865 tablet_copy_service.cc:161] P ce7fd35486204c12aef2ac2792ad5cec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d649133147ca40869e119ca7e2fe6710 at {username='oryx', principal='oryx/127.19.92.60@KRBTEST.COM'} at 127.0.0.1:51548: session id = d649133147ca40869e119ca7e2fe6710-00000000000000000000000000000000
I20260809 04:35:32.622051 28865 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 04:35:32.623432 29638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 04:35:32.625910 29638 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: tablet copy: Starting download of 0 data blocks...
I20260809 04:35:32.626111 29638 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: tablet copy: Starting download of 1 WAL segments...
I20260809 04:35:32.627543 29638 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 04:35:32.629387 28865 tablet_copy_service.cc:342] P ce7fd35486204c12aef2ac2792ad5cec: Request end of tablet copy session d649133147ca40869e119ca7e2fe6710-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.92.60@KRBTEST.COM'} at 127.0.0.1:51548
I20260809 04:35:32.629467 28865 tablet_copy_service.cc:433] P ce7fd35486204c12aef2ac2792ad5cec: ending tablet copy session d649133147ca40869e119ca7e2fe6710-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d649133147ca40869e119ca7e2fe6710
I20260809 04:35:32.637233 29638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:32.639047 29916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.639055 29912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.639703 29638 server_base.cc:1034] running on GCE node
W20260809 04:35:32.641292 29913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.641502 29638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:32.641817 29638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:32.642971 29638 hybrid_clock.cc:648] HybridClock initialized: now 1786250132642949 us; error 39 us; skew 500 ppm
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:32.645560 29638 init.cc:377] Logged in from keytab as oryx/127.19.92.60@KRBTEST.COM (short username oryx)
I20260809 04:35:32.646039 29638 webserver.cc:460] Webserver started at http://127.19.92.60:45559/ using document root <none> and password file <none>
I20260809 04:35:32.646219 29638 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:32.646261 29638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:32.647604 29638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:32.648705 29932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.648954 29638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:32.649027 29638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
uuid: "d649133147ca40869e119ca7e2fe6710"
format_stamp: "Formatted at 2026-08-09 04:35:32 on dist-test-slave-rmhg"
server_key: "5d620efe3f261c51a3cc30848cb292be"
server_key_iv: "7c140826eb18fce86d2369cb68081ea7"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.649103 29638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:32.667376 29638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:32.669716 29638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:32.674816 29638 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.60:33155
I20260809 04:35:32.675223 30008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.60:33155 every 8 connection(s)
I20260809 04:35:32.675628 29638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
I20260809 04:35:32.677829 30010 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:35:32.679034 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Bootstrap starting.
I20260809 04:35:32.690758 29292 heartbeater.cc:503] Master 127.19.92.62:34937 was elected leader, sending a full tablet report...
I20260809 04:35:32.691416 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:35:32.691730 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Bootstrap complete.
I20260809 04:35:32.692354 30010 raft_consensus.cc:348] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } }
I20260809 04:35:32.692503 30010 raft_consensus.cc:729] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d649133147ca40869e119ca7e2fe6710, State: Initialized, Role: LEARNER
I20260809 04:35:32.692615 30010 consensus_queue.cc:260] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } }
I20260809 04:35:32.692992 30016 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:32.693120 30010 sys_catalog.cc:564] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:32.693264 30016 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: This master's current role is: LEARNER
I20260809 04:35:32.696167 30031 catalog_manager.cc:1259] Loaded cluster ID: 1420b5688c154c7eb3506004a0575921
I20260809 04:35:32.696231 30031 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: loading cluster ID for follower catalog manager: success
I20260809 04:35:32.698458 30031 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: acquiring CA information for follower catalog manager: success
I20260809 04:35:32.699875 30031 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 04:35:32.701226 29888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:32.701468 29888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:32.701648 29888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:32.705902 29888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:32.705962 29888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:32.706038 29888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:32.711426 29888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab
--principal=oryx/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:45189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=36641
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731,127.19.92.60:33155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:32.712436 29888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:32.713469 29888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:35:32.723722 29888 server_base.cc:1034] running on GCE node
W20260809 04:35:32.723656 30034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.723692 30035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.723651 30037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.724295 29888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:32.724946 29888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:32.726111 29888 hybrid_clock.cc:648] HybridClock initialized: now 1786250132726079 us; error 45 us; skew 500 ppm
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:32.728789 29888 init.cc:377] Logged in from keytab as oryx/127.19.92.1@KRBTEST.COM (short username oryx)
I20260809 04:35:32.729745 29888 webserver.cc:460] Webserver started at http://127.19.92.1:36641/ using document root <none> and password file <none>
I20260809 04:35:32.730296 29888 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:32.730382 29888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:32.734292 29888 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:35:32.736353 30049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.737601 29888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:32.737721 29888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "2e4c93a7abd24a6492b3d1eb45c25149"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "873f5725554de9ef366d7c515c82cfce"
server_key_iv: "3093712298d34c0fcab01838e44e8e0a"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.738256 29888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:32.758512 29888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:32.761837 29888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:32.762028 29888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:32.762667 29888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:32.763597 29888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:32.763649 29888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.763689 29888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:32.763705 29888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.775056 29888 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:45189
I20260809 04:35:32.775104 30221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:45189 every 8 connection(s)
I20260809 04:35:32.776168 29888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:32.785398 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 29888
I20260809 04:35:32.785612 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 29115
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.61@KRBTEST.COM
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.60@KRBTEST.COM
I20260809 04:35:32.793735 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.778267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:44735 (local address 127.19.92.62:34937)
0809 04:35:32.778411 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.778416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.779769 (+ 1353us) server_negotiation.cc:407] Connection header received
0809 04:35:32.780840 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.780843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.780899 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.780961 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.782615 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.783425 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.785079 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.785408 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.785503 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.788872 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.788876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.788880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.788909 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.791238 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.791947 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.791951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.791953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.792058 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.792364 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.792367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.792369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.792606 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.792678 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.793426 (+ 748us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.793561 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:35:32.793784 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.778341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:35765 (local address 127.19.92.61:40731)
0809 04:35:32.778519 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.778523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.780213 (+ 1690us) server_negotiation.cc:407] Connection header received
0809 04:35:32.780840 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.780843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.780899 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.780953 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.782606 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.783158 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.785101 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.785428 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.785534 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.789053 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.789058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.789062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.789095 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.790778 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.791528 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.791532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.791535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.791645 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.792154 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.792158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.792160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.792444 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.792530 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.793455 (+ 925us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.793605 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260809 04:35:32.794898 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.778593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:44485 (local address 127.19.92.60:33155)
0809 04:35:32.779148 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.779152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.780248 (+ 1096us) server_negotiation.cc:407] Connection header received
0809 04:35:32.780806 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:32.780809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:32.780866 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:32.780945 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:32.782560 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.783689 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.784781 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.785077 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.785217 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.790640 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:32.790644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:32.790647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:32.790677 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:32.792255 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.792722 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.792726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.792727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.792819 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:32.793314 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:32.793317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:32.793319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:32.793555 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:32.793626 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:32.794612 (+ 986us) server_negotiation.cc:299] Negotiation successful
0809 04:35:32.794739 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":75,"threads_started":1}
I20260809 04:35:32.795955 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.778953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.19.92.1:35765)
0809 04:35:32.780103 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.780116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.780611 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.781215 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.781218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.781757 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.782457 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.782465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.783368 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.783371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.784952 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.784965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.785664 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.785668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.785778 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.786457 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.786480 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.788853 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.790947 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.790955 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.790975 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.791393 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.791776 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.791779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.791781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.792021 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.792624 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.792632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.793070 (+ 438us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.793985 (+ 915us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.794464 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":245,"spinlock_wait_cycles":35072,"thread_start_us":233,"threads_started":1}
I20260809 04:35:32.796303 30234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.778953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.19.92.1:44485)
0809 04:35:32.780151 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.780163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.780609 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.781138 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.781146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.781747 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.782427 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.782444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.783787 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.783791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.784661 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.784672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.785384 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.785389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.785664 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.788762 (+ 3098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.788776 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.790432 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.792364 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.792366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.792368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.792606 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.792922 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.792925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.792927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.793020 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.793981 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.793984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.794152 (+ 168us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.795308 (+ 1156us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.795455 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":2783,"spinlock_wait_cycles":4096,"thread_start_us":267,"threads_started":1}
I20260809 04:35:32.797192 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.778953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:44735)
0809 04:35:32.779608 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.779661 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:32.780611 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:32.781139 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:32.781174 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:32.781752 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:32.782454 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.782465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.783535 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.783538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.784960 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:32.784966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:32.785845 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:32.785849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:32.785925 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:32.786533 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:32.786546 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:32.788662 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:32.791372 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.791375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.791377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.791830 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.792176 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:32.792179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:32.792182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:32.792266 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:32.792743 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:32.792745 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:32.793062 (+ 317us) client_negotiation.cc:769] Sending connection context
0809 04:35:32.794467 (+ 1405us) client_negotiation.cc:240] Negotiation successful
0809 04:35:32.794616 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":187,"spinlock_wait_cycles":3712,"thread_start_us":297,"threads_started":1}
I20260809 04:35:32.797986 30222 heartbeater.cc:347] Connected to a master server at 127.19.92.60:33155
I20260809 04:35:32.798130 30224 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34937
I20260809 04:35:32.798297 30224 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:32.798301 30222 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:32.798913 30224 heartbeater.cc:511] Master 127.19.92.62:34937 requested a full tablet report, sending...
I20260809 04:35:32.798925 30222 heartbeater.cc:511] Master 127.19.92.60:33155 requested a full tablet report, sending...
I20260809 04:35:32.800294 28831 ts_manager.cc:194] Re-registered known tserver with Master: 2e4c93a7abd24a6492b3d1eb45c25149 (127.19.92.1:45189)
I20260809 04:35:32.800691 29943 ts_manager.cc:194] Registered new tserver with Master: 2e4c93a7abd24a6492b3d1eb45c25149 (127.19.92.1:45189)
I20260809 04:35:32.801079 28831 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:44735
I20260809 04:35:32.807523 30226 heartbeater.cc:347] Connected to a master server at 127.19.92.61:40731
I20260809 04:35:32.807636 30226 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:32.807859 30226 heartbeater.cc:511] Master 127.19.92.61:40731 requested a full tablet report, sending...
I20260809 04:35:32.809019 29693 ts_manager.cc:194] Registered new tserver with Master: 2e4c93a7abd24a6492b3d1eb45c25149 (127.19.92.1:45189)
I20260809 04:35:32.813357 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:45401
--local_ip_for_outbound_sockets=127.19.92.2
--tserver_master_addrs=127.19.92.62:34937
--webserver_port=39083
--webserver_interface=127.19.92.2
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab
--principal=oryx/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731,127.19.92.60:33155 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:35:32.922086 29479 heartbeater.cc:503] Master 127.19.92.62:34937 was elected leader, sending a full tablet report...
W20260809 04:35:32.930027 30236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:32.930279 30236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:32.930325 30236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:32.934337 30236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:32.934399 30236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:32.934465 30236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:32.939786 30236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab
--principal=oryx/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:45401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=39083
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731,127.19.92.60:33155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:32.940863 30236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:32.941973 30236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:32.951735 30247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.951754 30243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:32.951754 30245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:32.952042 30236 server_base.cc:1034] running on GCE node
I20260809 04:35:32.952435 30236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:32.953059 30236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:32.954213 30236 hybrid_clock.cc:648] HybridClock initialized: now 1786250132954193 us; error 31 us; skew 500 ppm
Aug 09 04:35:32 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:32.956895 30236 init.cc:377] Logged in from keytab as oryx/127.19.92.2@KRBTEST.COM (short username oryx)
I20260809 04:35:32.957892 30236 webserver.cc:460] Webserver started at http://127.19.92.2:39083/ using document root <none> and password file <none>
I20260809 04:35:32.958488 30236 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:32.958577 30236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:32.962692 30236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:35:32.964694 30255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.965929 30236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:32.966055 30236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "404364fda6b846dcb4dedb42b23e7caf"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "38e5d42751a512e206ff2a09a73e4d9e"
server_key_iv: "2e921ffc2062a90d9ff007d66b4d07f6"
server_key_version: "encryptionkey@0"
I20260809 04:35:32.966586 30236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:32.980563 30236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:32.983354 30236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:32.983525 30236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:32.984123 30236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:32.984997 30236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:32.985042 30236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.985082 30236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:32.985110 30236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:32.994772 30236 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:45401
I20260809 04:35:32.994817 30426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:45401 every 8 connection(s)
I20260809 04:35:32.995896 30236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:33.000474 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 30236
I20260809 04:35:33.000686 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 29297
Aug 09 04:35:33 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.60@KRBTEST.COM
I20260809 04:35:33.019831 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.998612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:35123 (local address 127.19.92.60:33155)
0809 04:35:32.998801 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.998806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.999769 (+ 963us) server_negotiation.cc:407] Connection header received
0809 04:35:33.000517 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.000521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.000582 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.000636 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.010800 (+ 10164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.011494 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.012951 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.013235 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.013335 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.016131 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.016135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.016137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.016167 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.017623 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.018278 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.018282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.018283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.018364 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.018681 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.018684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.018686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.018880 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.018940 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.019547 (+ 607us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.019678 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 04:35:33.021627 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.998467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.19.92.2:35123)
0809 04:35:32.999640 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.999652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.000228 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.009119 (+ 8891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.009129 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.009764 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:33.010594 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.010618 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.011649 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.011651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.012813 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.012824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.013395 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.013399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.013646 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.014293 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:33.014310 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:33.015942 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:33.017742 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.017748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.017763 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.018174 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.018449 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.018451 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.018453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.018598 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.019006 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:33.019012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:33.019301 (+ 289us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.020042 (+ 741us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.020454 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":157,"threads_started":1}
I20260809 04:35:33.023221 30427 heartbeater.cc:347] Connected to a master server at 127.19.92.60:33155
I20260809 04:35:33.023447 30427 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:33.023947 30427 heartbeater.cc:511] Master 127.19.92.60:33155 requested a full tablet report, sending...
I20260809 04:35:33.025019 29943 ts_manager.cc:194] Registered new tserver with Master: 404364fda6b846dcb4dedb42b23e7caf (127.19.92.2:45401)
Aug 09 04:35:33 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.61@KRBTEST.COM
Aug 09 04:35:33 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250132, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:33.041276 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:45393
--local_ip_for_outbound_sockets=127.19.92.3
--tserver_master_addrs=127.19.92.62:34937
--webserver_port=36129
--webserver_interface=127.19.92.3
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab
--principal=oryx/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731,127.19.92.60:33155 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:35:33.043942 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.998178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:42721 (local address 127.19.92.62:34937)
0809 04:35:32.998339 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.998344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.999983 (+ 1639us) server_negotiation.cc:407] Connection header received
0809 04:35:33.000504 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.000508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.000557 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.000607 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.035801 (+ 35194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.036609 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.037913 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.038251 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.038356 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.040679 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.040683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.040685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.040718 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.042170 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.042616 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.042619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.042621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.042716 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.043036 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.043040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.043041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.043297 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.043361 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.043672 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.043792 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:35:33.043954 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.998467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:42721)
0809 04:35:32.999895 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.999904 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.000236 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.035251 (+ 35015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.035253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.035475 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:33.035633 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.035641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.036718 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.036721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.037765 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.037776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.038422 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.038426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.038502 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.038908 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:33.038920 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:33.040544 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:33.042288 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.042290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.042292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.042526 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.042808 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.042810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.042812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.042905 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.043421 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:33.043423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:33.043499 (+ 76us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.043585 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.043736 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"mutex_wait_us":82,"spinlock_wait_cycles":26752,"thread_start_us":176,"threads_started":1}
I20260809 04:35:33.044026 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.998774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:47063 (local address 127.19.92.61:40731)
0809 04:35:32.998903 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:32.998907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:32.999339 (+ 432us) server_negotiation.cc:407] Connection header received
0809 04:35:33.000486 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.000490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.000550 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.000607 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.034954 (+ 34347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.035746 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.036647 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.037000 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.037111 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.040380 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.040384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.040387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.040418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.042110 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.042654 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.042657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.042659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.042744 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.043069 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.043073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.043075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.043299 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.043370 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.043725 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.043865 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:35:33.044562 30428 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34937
I20260809 04:35:33.044678 30428 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:33.044867 30428 heartbeater.cc:511] Master 127.19.92.62:34937 requested a full tablet report, sending...
I20260809 04:35:33.044950 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:32.998466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.19.92.2:47063)
0809 04:35:32.999169 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:32.999228 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.000223 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.034188 (+ 33965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.034190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.034558 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:33.034833 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.034843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.035867 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.035871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.036529 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.036538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.037127 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.037130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.037315 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.038535 (+ 1220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:33.038567 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:33.040203 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:33.042236 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.042240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.042242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.042526 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.042850 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.042853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.042855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.042948 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.043441 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:33.043443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:33.043551 (+ 108us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.044236 (+ 685us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.044377 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":32,"spinlock_wait_cycles":470656,"thread_start_us":297,"threads_started":1}
I20260809 04:35:33.045557 28831 ts_manager.cc:194] Re-registered known tserver with Master: 404364fda6b846dcb4dedb42b23e7caf (127.19.92.2:45401)
I20260809 04:35:33.045805 30430 heartbeater.cc:347] Connected to a master server at 127.19.92.61:40731
I20260809 04:35:33.045902 30430 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:33.046069 30430 heartbeater.cc:511] Master 127.19.92.61:40731 requested a full tablet report, sending...
I20260809 04:35:33.046290 28831 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:42721
I20260809 04:35:33.046801 29693 ts_manager.cc:194] Registered new tserver with Master: 404364fda6b846dcb4dedb42b23e7caf (127.19.92.2:45401)
W20260809 04:35:33.159145 30437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:33.159438 30437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:33.159484 30437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:33.163587 30437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:33.163653 30437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:33.163717 30437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:33.169111 30437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab
--principal=oryx/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:45393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=36129
--tserver_master_addrs=127.19.92.62:34937,127.19.92.61:40731,127.19.92.60:33155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:33.170290 30437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:33.171353 30437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:33.181066 30448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:33.181250 30446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:33.181401 30452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:33.181505 30437 server_base.cc:1034] running on GCE node
I20260809 04:35:33.181860 30437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:33.182541 30437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:33.183701 30437 hybrid_clock.cc:648] HybridClock initialized: now 1786250133183670 us; error 45 us; skew 500 ppm
Aug 09 04:35:33 dist-test-slave-rmhg krb5kdc[28769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250133, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:33.186610 30437 init.cc:377] Logged in from keytab as oryx/127.19.92.3@KRBTEST.COM (short username oryx)
I20260809 04:35:33.187588 30437 webserver.cc:460] Webserver started at http://127.19.92.3:36129/ using document root <none> and password file <none>
I20260809 04:35:33.188197 30437 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:33.188284 30437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:33.193004 30437 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:33.195156 30462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:33.196537 30437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:33.196669 30437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "4e157049549f428d8798db9612b2b72c"
format_stamp: "Formatted at 2026-08-09 04:35:31 on dist-test-slave-rmhg"
server_key: "fc8ad567a94428c44ac67de16108eb9b"
server_key_iv: "870e11181f1c88f16cd0243cfb72d448"
server_key_version: "encryptionkey@0"
I20260809 04:35:33.197400 30437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:33.206616 30437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:33.210343 30437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:33.210507 30437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:33.211115 30437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:33.212163 30437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:33.212175 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.208266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.0.0.1:37086)
0809 04:35:33.208501 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.208514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.208619 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.208904 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.208906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.208918 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:33.209030 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.209035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.210050 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.210054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.210886 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.210898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.211790 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.211793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.211888 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.211919 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.211991 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.212057 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260809 04:35:33.212216 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.208359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.19.92.60:33155)
0809 04:35:33.208479 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.208483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.208593 (+ 110us) server_negotiation.cc:407] Connection header received
0809 04:35:33.208717 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.208721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.208789 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.208895 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:33.209135 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.209930 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.211040 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.211676 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.211793 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.211957 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.212016 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.212092 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:35:33.212222 30437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:33.212275 30437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:33.212296 30437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:33.212814 29971 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 04:35:33.213383 30476 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7fd35486204c12aef2ac2792ad5cec. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:33.213670 30476 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: This master's current role is: LEARNER
I20260809 04:35:33.222424 30437 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:45393
I20260809 04:35:33.222508 30638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:45393 every 8 connection(s)
I20260809 04:35:33.223413 30437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:33.230202 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 30437
Aug 09 04:35:33 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250133, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
Aug 09 04:35:33 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250133, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.60@KRBTEST.COM
Aug 09 04:35:33 dist-test-slave-rmhg krb5kdc[28769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250133, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.61@KRBTEST.COM
I20260809 04:35:33.243745 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.225481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:52375 (local address 127.19.92.60:33155)
0809 04:35:33.225585 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.225588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.227180 (+ 1592us) server_negotiation.cc:407] Connection header received
0809 04:35:33.228176 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.228180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.228242 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.228314 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.229623 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.230435 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.233369 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.233690 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.233766 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.237920 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.237925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.237928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.237978 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.239389 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.240355 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.240360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.240362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.240471 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.242709 (+ 2238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.242712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.242714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.242972 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.243044 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.243477 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.243591 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":36}
I20260809 04:35:33.244024 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.225796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.19.92.3:52375)
0809 04:35:33.227031 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.227084 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.227979 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.228488 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.228497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.228892 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:33.229487 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.229503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.232317 (+ 2814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.232319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.233225 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.233235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.234396 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.234401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.234568 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.235105 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:33.235120 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:33.237673 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:33.239517 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.239525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.239549 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.239921 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.241963 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.241966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.241968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.242056 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.243219 (+ 1163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:33.243221 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:33.243302 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.243400 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.243552 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":82,"spinlock_wait_cycles":69632,"thread_start_us":254,"threads_started":1}
I20260809 04:35:33.244339 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.225796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:56215)
0809 04:35:33.227033 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.227081 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.227986 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.228489 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.228498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.228914 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:33.229488 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.229505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.231223 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.231226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.231960 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.231972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.233342 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.233345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.233545 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.234349 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:33.234371 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:33.237486 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:33.240124 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.240127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.240129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.240369 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.240711 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.240714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.240716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.240946 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.241546 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:33.241555 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:33.241969 (+ 414us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.242311 (+ 342us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.242736 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":152,"spinlock_wait_cycles":7552,"thread_start_us":1338,"threads_started":1}
I20260809 04:35:33.244796 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.226147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:47851 (local address 127.19.92.61:40731)
0809 04:35:33.226279 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.226284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.227646 (+ 1362us) server_negotiation.cc:407] Connection header received
0809 04:35:33.228220 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.228223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.228269 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.228317 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.229660 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.230434 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.232814 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.233076 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.233193 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.239504 (+ 6311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.239509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.239511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.239545 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.241230 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.241782 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.241786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.241788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.241878 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.242321 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.242325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.242327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.242552 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.242619 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.244499 (+ 1880us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.244647 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:35:33.245090 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.225244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:56215 (local address 127.19.92.62:34937)
0809 04:35:33.225369 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.225374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.227187 (+ 1813us) server_negotiation.cc:407] Connection header received
0809 04:35:33.228203 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.228206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.228261 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.228313 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.230570 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.231090 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.232114 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.232424 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.232509 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.237668 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.237672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.237675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.237706 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.239743 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.240481 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.240484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.240486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.240601 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.241067 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.241071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.241073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.241340 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.241437 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.244821 (+ 3384us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.244942 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:35:33.245298 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.226031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.19.92.3:47851)
0809 04:35:33.227513 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.227526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.227986 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.228489 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.228498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.228914 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:33.229488 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.229505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.230644 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.230648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.232197 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.232208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.233247 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.233251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.233560 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.237754 (+ 4194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:33.237768 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:33.239324 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:33.241353 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.241356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.241358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.241668 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.242152 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:33.242154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:33.242156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:33.242224 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:33.242990 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:33.242993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:33.243087 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.244982 (+ 1895us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.245121 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"mutex_wait_us":142,"spinlock_wait_cycles":18304,"thread_start_us":818,"threads_started":1}
I20260809 04:35:33.246047 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.232662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.19.92.62:34937)
0809 04:35:33.232876 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.232880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.233620 (+ 740us) server_negotiation.cc:407] Connection header received
0809 04:35:33.233660 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.233663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.233709 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.233836 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.234915 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.235759 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.236724 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.237207 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.237307 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.241933 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.241936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.241939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.241962 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.243700 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.244207 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.244212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.244213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.244311 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.244659 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.244662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.244664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.244883 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.245004 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.245805 (+ 801us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.245926 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260809 04:35:33.246133 30642 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34937
I20260809 04:35:33.246341 30639 heartbeater.cc:347] Connected to a master server at 127.19.92.60:33155
I20260809 04:35:33.246408 30642 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:33.246435 30639 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:33.247016 30642 heartbeater.cc:511] Master 127.19.92.62:34937 requested a full tablet report, sending...
I20260809 04:35:33.247013 30639 heartbeater.cc:511] Master 127.19.92.60:33155 requested a full tablet report, sending...
I20260809 04:35:33.247087 30643 heartbeater.cc:347] Connected to a master server at 127.19.92.61:40731
I20260809 04:35:33.247264 30643 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:33.247401 30643 heartbeater.cc:511] Master 127.19.92.61:40731 requested a full tablet report, sending...
I20260809 04:35:33.248200 28831 ts_manager.cc:194] Re-registered known tserver with Master: 4e157049549f428d8798db9612b2b72c (127.19.92.3:45393)
I20260809 04:35:33.248395 29693 ts_manager.cc:194] Registered new tserver with Master: 4e157049549f428d8798db9612b2b72c (127.19.92.3:45393)
I20260809 04:35:33.248626 29943 ts_manager.cc:194] Registered new tserver with Master: 4e157049549f428d8798db9612b2b72c (127.19.92.3:45393)
I20260809 04:35:33.248919 28831 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:56215
I20260809 04:35:33.253238 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.237066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.19.92.60:33155)
0809 04:35:33.238043 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.238048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.238562 (+ 514us) server_negotiation.cc:407] Connection header received
0809 04:35:33.238608 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.238610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.238667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.238833 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.240371 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.241230 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.242389 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.242787 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.242872 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.249782 (+ 6910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.249785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.249788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.249813 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.251465 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.251941 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.251945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.251947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.252035 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.252310 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.252313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.252315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.252553 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.252678 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.252909 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.253047 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":902,"thread_start_us":114,"threads_started":1}
I20260809 04:35:33.250326 28830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:33.254510 28830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:33.255812 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.232279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.19.92.61:40731)
0809 04:35:33.233093 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.233098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.234251 (+ 1153us) server_negotiation.cc:407] Connection header received
0809 04:35:33.234295 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.234298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.234346 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.234585 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.235570 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.236435 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.250167 (+ 13732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.250650 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.250754 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.252196 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.252200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.252203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.252228 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.254263 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.254657 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.254660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.254662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.254751 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.254977 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.254980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.254982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.255197 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.255322 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.255526 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.255655 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":75,"threads_started":1}
I20260809 04:35:33.257645 29902 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: NON_VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: true } } }
I20260809 04:35:33.257769 29902 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:33.261406 30668 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: attempting to promote NON_VOTER d649133147ca40869e119ca7e2fe6710 to VOTER
W20260809 04:35:33.261682 30666 consensus_queue.cc:1159] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e1c9cc2ed7e8400186869797c90d58a9" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20260809 04:35:33.261948 30668 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } }
I20260809 04:35:33.264973 29971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 LEARNER]: Refusing update from remote peer ce7fd35486204c12aef2ac2792ad5cec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 04:35:33.264993 29705 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Refusing update from remote peer ce7fd35486204c12aef2ac2792ad5cec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 04:35:33.265393 30666 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 04:35:33.265638 29902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 04:35:33.267256 30666 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d649133147ca40869e119ca7e2fe6710 (127.19.92.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:33.267694 29705 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d649133147ca40869e119ca7e2fe6710 (127.19.92.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:33.267930 29971 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d649133147ca40869e119ca7e2fe6710 (127.19.92.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:33.268983 30666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:33.269106 30666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:33.269254 30666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:33.269345 30666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:33.269467 30666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:33.269567 30666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:33.269447 30682 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:33.269663 30682 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:35:33.270186 30476 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce7fd35486204c12aef2ac2792ad5cec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:33.270296 30476 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:35:33.277889 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.271853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:45401 (local address 127.0.0.1:51506)
0809 04:35:33.271997 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.272014 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.272101 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.273469 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.273473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.273492 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:33.273611 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.273617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.275010 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.275014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.276219 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.276231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.277431 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.277435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.277533 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.277562 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.277656 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.277740 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 04:35:33.278187 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.272014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.19.92.2:45401)
0809 04:35:33.272687 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.272726 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.272753 (+ 27us) server_negotiation.cc:407] Connection header received
0809 04:35:33.272840 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.272846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.273123 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.273514 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:33.273736 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.274880 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.276390 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.277302 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.277427 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.277600 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.277696 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.277805 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":179,"threads_started":1}
I20260809 04:35:33.278265 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.272385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:45393 (local address 127.0.0.1:42428)
0809 04:35:33.273724 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.273737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.274032 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.274564 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.274567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.274587 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:33.274682 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.274688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.275791 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.275795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.276684 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.276696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.277936 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.277940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.278032 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.278058 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.278106 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.278160 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260809 04:35:33.278867 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.272846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.19.92.3:45393)
0809 04:35:33.273251 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.273283 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.273994 (+ 711us) server_negotiation.cc:407] Connection header received
0809 04:35:33.274132 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.274137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.274301 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.274580 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:33.274773 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.275685 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.276818 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.277807 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.278437 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.278615 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.278670 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.278757 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260809 04:35:33.279469 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.272077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45189 (local address 127.0.0.1:46930)
0809 04:35:33.272325 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.272337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.272642 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.274983 (+ 2341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.274986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.275005 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:33.275095 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.275101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.276437 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.276440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.277600 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.277616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.279086 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.279090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.279180 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.279204 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.279273 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.279345 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":119,"threads_started":1}
I20260809 04:35:33.280087 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.272538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.19.92.1:45189)
0809 04:35:33.274266 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.274320 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.274342 (+ 22us) server_negotiation.cc:407] Connection header received
0809 04:35:33.274418 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.274423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.274620 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.275003 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:33.275187 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.276318 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.277741 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.278765 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.278868 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.279017 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.279689 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.279815 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":78,"threads_started":1}
I20260809 04:35:33.281554 30335 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b11822fa04fb58b3dc4aba14d12e4 (DEFAULT_TABLE table=test-table [id=e8bb2d0d67a14ff58cbf890fafb05c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:33.282577 30335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b11822fa04fb58b3dc4aba14d12e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:33.282189 30542 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b11822fa04fb58b3dc4aba14d12e4 (DEFAULT_TABLE table=test-table [id=e8bb2d0d67a14ff58cbf890fafb05c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:33.282775 30128 tablet_service.cc:1467] Processing CreateTablet for tablet ad3b11822fa04fb58b3dc4aba14d12e4 (DEFAULT_TABLE table=test-table [id=e8bb2d0d67a14ff58cbf890fafb05c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:33.283288 30542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b11822fa04fb58b3dc4aba14d12e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:33.283805 30128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3b11822fa04fb58b3dc4aba14d12e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:33.291623 30690 tablet_bootstrap.cc:492] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: Bootstrap starting.
I20260809 04:35:33.292481 30692 tablet_bootstrap.cc:492] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: Bootstrap starting.
I20260809 04:35:33.293785 30693 tablet_bootstrap.cc:492] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: Bootstrap starting.
I20260809 04:35:33.295210 30692 tablet_bootstrap.cc:654] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:33.295255 30690 tablet_bootstrap.cc:654] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:33.296190 30693 tablet_bootstrap.cc:654] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:33.296675 30690 log.cc:824] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:33.296730 30692 log.cc:824] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:33.297503 30693 log.cc:824] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:33.298991 30692 tablet_bootstrap.cc:492] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: No bootstrap required, opened a new log
I20260809 04:35:33.298995 30690 tablet_bootstrap.cc:492] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: No bootstrap required, opened a new log
I20260809 04:35:33.299190 30692 ts_tablet_manager.cc:1397] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 04:35:33.299194 30690 ts_tablet_manager.cc:1397] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 04:35:33.299671 30693 tablet_bootstrap.cc:492] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: No bootstrap required, opened a new log
I20260809 04:35:33.299857 30693 ts_tablet_manager.cc:1397] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:35:33.304975 30693 raft_consensus.cc:348] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.305316 30693 raft_consensus.cc:374] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:33.305397 30693 raft_consensus.cc:729] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4c93a7abd24a6492b3d1eb45c25149, State: Initialized, Role: FOLLOWER
I20260809 04:35:33.305326 30692 raft_consensus.cc:348] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.305660 30692 raft_consensus.cc:374] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:33.305717 30692 raft_consensus.cc:729] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e157049549f428d8798db9612b2b72c, State: Initialized, Role: FOLLOWER
I20260809 04:35:33.306006 30693 consensus_queue.cc:260] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.306175 30692 consensus_queue.cc:260] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.306032 30690 raft_consensus.cc:348] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.306335 30690 raft_consensus.cc:374] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:33.306385 30690 raft_consensus.cc:729] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404364fda6b846dcb4dedb42b23e7caf, State: Initialized, Role: FOLLOWER
I20260809 04:35:33.306836 30690 consensus_queue.cc:260] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.307018 30693 ts_tablet_manager.cc:1428] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260809 04:35:33.307349 30224 heartbeater.cc:503] Master 127.19.92.62:34937 was elected leader, sending a full tablet report...
I20260809 04:35:33.308563 30690 ts_tablet_manager.cc:1428] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20260809 04:35:33.309846 30692 ts_tablet_manager.cc:1428] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260809 04:35:33.310387 30642 heartbeater.cc:503] Master 127.19.92.62:34937 was elected leader, sending a full tablet report...
I20260809 04:35:33.311895 30428 heartbeater.cc:503] Master 127.19.92.62:34937 was elected leader, sending a full tablet report...
I20260809 04:35:33.315928 30701 raft_consensus.cc:482] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:33.316129 30701 raft_consensus.cc:504] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.317539 30701 leader_election.cc:288] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 404364fda6b846dcb4dedb42b23e7caf (127.19.92.2:45401), 4e157049549f428d8798db9612b2b72c (127.19.92.3:45393)
I20260809 04:35:33.322780 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.317878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:35229 (local address 127.19.92.2:45401)
0809 04:35:33.317989 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.317993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.318300 (+ 307us) server_negotiation.cc:407] Connection header received
0809 04:35:33.318546 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.318550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.318601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.318676 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:33.319184 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.319939 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.321562 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.322156 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.322272 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.322342 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.322596 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.322671 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:35:33.322873 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.317760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:45401 (local address 127.19.92.1:35229)
0809 04:35:33.318214 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.318229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.318373 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.318786 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.318789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.318807 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:33.319066 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.319072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.320077 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.320081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.321441 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.321453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.322287 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.322291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.322475 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.322496 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.322564 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.322640 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":15,"spinlock_wait_cycles":8832,"thread_start_us":91,"threads_started":1}
I20260809 04:35:33.324409 30363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3b11822fa04fb58b3dc4aba14d12e4" candidate_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404364fda6b846dcb4dedb42b23e7caf" is_pre_election: true
I20260809 04:35:33.324638 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.317927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:43347 (local address 127.19.92.3:45393)
0809 04:35:33.318056 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.318060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.318562 (+ 502us) server_negotiation.cc:407] Connection header received
0809 04:35:33.318664 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.318667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.318713 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.318779 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:33.320102 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.320831 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.322787 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.323498 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.323572 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.323641 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.324383 (+ 742us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.324525 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 04:35:33.324836 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.317760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:45393 (local address 127.19.92.1:43347)
0809 04:35:33.318411 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:33.318425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:33.318580 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:33.318795 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:33.318798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:33.318823 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:33.319061 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.319071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.320966 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.320969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.322641 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:33.322652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.324077 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.324080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:33.324244 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.324267 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:35:33.324335 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 04:35:33.324401 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":317,"spinlock_wait_cycles":4224,"thread_start_us":166,"threads_started":1}
I20260809 04:35:33.324793 30363 raft_consensus.cc:2454] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4c93a7abd24a6492b3d1eb45c25149 in term 0.
I20260809 04:35:33.325590 30579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3b11822fa04fb58b3dc4aba14d12e4" candidate_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e157049549f428d8798db9612b2b72c" is_pre_election: true
I20260809 04:35:33.325877 30579 raft_consensus.cc:2454] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4c93a7abd24a6492b3d1eb45c25149 in term 0.
I20260809 04:35:33.326463 30055 leader_election.cc:302] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4c93a7abd24a6492b3d1eb45c25149, 4e157049549f428d8798db9612b2b72c; no voters:
I20260809 04:35:33.326897 30701 raft_consensus.cc:2781] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:33.326998 30701 raft_consensus.cc:482] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:33.327039 30701 raft_consensus.cc:3037] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:33.328753 30701 raft_consensus.cc:504] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.329089 30701 leader_election.cc:288] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [CANDIDATE]: Term 1 election: Requested vote from peers 404364fda6b846dcb4dedb42b23e7caf (127.19.92.2:45401), 4e157049549f428d8798db9612b2b72c (127.19.92.3:45393)
I20260809 04:35:33.329492 30363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3b11822fa04fb58b3dc4aba14d12e4" candidate_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404364fda6b846dcb4dedb42b23e7caf"
I20260809 04:35:33.329648 30363 raft_consensus.cc:3037] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:33.329674 30579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3b11822fa04fb58b3dc4aba14d12e4" candidate_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e157049549f428d8798db9612b2b72c"
I20260809 04:35:33.329798 30579 raft_consensus.cc:3037] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:33.331681 30579 raft_consensus.cc:2454] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4c93a7abd24a6492b3d1eb45c25149 in term 1.
I20260809 04:35:33.331681 30363 raft_consensus.cc:2454] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4c93a7abd24a6492b3d1eb45c25149 in term 1.
I20260809 04:35:33.331990 30053 leader_election.cc:302] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4c93a7abd24a6492b3d1eb45c25149, 404364fda6b846dcb4dedb42b23e7caf; no voters:
I20260809 04:35:33.332176 30701 raft_consensus.cc:2781] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:33.332477 30701 raft_consensus.cc:686] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4c93a7abd24a6492b3d1eb45c25149, State: Running, Role: LEADER
I20260809 04:35:33.332871 30701 consensus_queue.cc:237] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:33.336517 28830 catalog_manager.cc:5495] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e4c93a7abd24a6492b3d1eb45c25149 (127.19.92.1). New cstate: current_term: 1 leader_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } health_report { overall_health: UNKNOWN } } }
I20260809 04:35:33.365129 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.360964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.19.92.1:45189)
0809 04:35:33.361116 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.361120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.361136 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:35:33.361342 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.361346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.361428 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.361537 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:33.361813 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.362614 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.363612 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.363851 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.363955 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.364273 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:33.364726 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:33.364841 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.364921 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.364985 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:35:33.372105 30579 raft_consensus.cc:1261] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 1 FOLLOWER]: Refusing update from remote peer 2e4c93a7abd24a6492b3d1eb45c25149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:33.372135 30363 raft_consensus.cc:1261] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 1 FOLLOWER]: Refusing update from remote peer 2e4c93a7abd24a6492b3d1eb45c25149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:33.372876 30710 consensus_queue.cc:1035] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:33.373067 30701 consensus_queue.cc:1035] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:33.383801 30714 mvcc.cc:205] Tried to move back new op lower bound from 7316480546283999232 to 7316480546134380544. Current Snapshot: MvccSnapshot[applied={T|T < 7316480546283999232 or (T in {7316480546283999232})}]
I20260809 04:35:33.386668 30715 mvcc.cc:205] Tried to move back new op lower bound from 7316480546283999232 to 7316480546134380544. Current Snapshot: MvccSnapshot[applied={T|T < 7316480546283999232}]
I20260809 04:35:33.389030 30719 mvcc.cc:205] Tried to move back new op lower bound from 7316480546283999232 to 7316480546134380544. Current Snapshot: MvccSnapshot[applied={T|T < 7316480546283999232}]
I20260809 04:35:33.392071 28829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:33.392298 28829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:33.396582 28829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce7fd35486204c12aef2ac2792ad5cec: Sending DeleteTablet for 3 replicas of tablet ad3b11822fa04fb58b3dc4aba14d12e4
I20260809 04:35:33.397550 30335 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3b11822fa04fb58b3dc4aba14d12e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:33 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:51506
I20260809 04:35:33.397761 30128 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3b11822fa04fb58b3dc4aba14d12e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:33 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:46930
I20260809 04:35:33.398201 30726 tablet_replica.cc:330] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: stopping tablet replica
I20260809 04:35:33.398538 30727 tablet_replica.cc:330] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: stopping tablet replica
I20260809 04:35:33.398698 30726 raft_consensus.cc:2229] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:33.398885 30727 raft_consensus.cc:2229] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:33.398984 30542 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3b11822fa04fb58b3dc4aba14d12e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:33 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:42428
I20260809 04:35:33.399240 30726 raft_consensus.cc:2258] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:33.399292 30727 raft_consensus.cc:2258] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:33.400892 30728 tablet_replica.cc:330] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: stopping tablet replica
I20260809 04:35:33.401257 30728 raft_consensus.cc:2229] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:33.401597 30728 raft_consensus.cc:2258] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:33.402413 30726 ts_tablet_manager.cc:1905] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:33.402779 30727 ts_tablet_manager.cc:1905] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:33.403087 30728 ts_tablet_manager.cc:1905] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:33.403219 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 28788
I20260809 04:35:33.415127 30726 ts_tablet_manager.cc:1918] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:33.415413 30728 ts_tablet_manager.cc:1918] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:33.415552 30728 log.cc:1196] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/ad3b11822fa04fb58b3dc4aba14d12e4
I20260809 04:35:33.415896 30728 ts_tablet_manager.cc:1939] T ad3b11822fa04fb58b3dc4aba14d12e4 P 4e157049549f428d8798db9612b2b72c: Deleting consensus metadata
I20260809 04:35:33.417513 30726 log.cc:1196] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/ad3b11822fa04fb58b3dc4aba14d12e4
I20260809 04:35:33.417865 30726 ts_tablet_manager.cc:1939] T ad3b11822fa04fb58b3dc4aba14d12e4 P 404364fda6b846dcb4dedb42b23e7caf: Deleting consensus metadata
I20260809 04:35:33.418744 30727 ts_tablet_manager.cc:1918] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:33.418898 30727 log.cc:1196] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/ad3b11822fa04fb58b3dc4aba14d12e4
I20260809 04:35:33.419229 30727 ts_tablet_manager.cc:1939] T ad3b11822fa04fb58b3dc4aba14d12e4 P 2e4c93a7abd24a6492b3d1eb45c25149: Deleting consensus metadata
W20260809 04:35:33.422607 30055 connection.cc:535] server connection from 127.0.0.1:46930 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 04:35:33.422626 30466 connection.cc:535] server connection from 127.0.0.1:42428 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 04:35:33.423007 30257 connection.cc:535] server connection from 127.0.0.1:51506 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 04:35:33.430908 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.401618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.19.92.61:40731)
0809 04:35:33.401879 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.401884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.421562 (+ 19678us) server_negotiation.cc:407] Connection header received
0809 04:35:33.421808 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.421812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.421875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.421947 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.422909 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.423659 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.424856 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.425220 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.425305 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.427345 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.427349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.427352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.427379 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.429004 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.429505 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.429509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.429511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.429597 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.429889 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.429892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.429894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.430114 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.430215 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.430624 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.430745 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260809 04:35:33.436440 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.426792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.19.92.60:33155)
0809 04:35:33.426907 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.426912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.427122 (+ 210us) server_negotiation.cc:407] Connection header received
0809 04:35:33.427316 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.427319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.427375 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.427432 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.428253 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.428998 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.430117 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.430459 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.430539 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.432425 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.432430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.432432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.432460 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.434504 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.434998 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.435002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.435003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.435095 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.435389 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.435393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.435394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.435625 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.435743 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.436113 (+ 370us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.436241 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:35:33.436458 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.426588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.19.92.61:40731)
0809 04:35:33.426700 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.426705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.427461 (+ 756us) server_negotiation.cc:407] Connection header received
0809 04:35:33.427616 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.427619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.427665 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.427731 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.428872 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.429637 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.430757 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.431070 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.431144 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.432604 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.432608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.432610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.432635 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.434546 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.435100 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.435103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.435105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.435193 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.435484 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.435488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.435490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.435702 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.435800 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.436177 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.436299 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:33.440816 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:33.401744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.19.92.60:33155)
0809 04:35:33.401969 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:33.401973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:33.432838 (+ 30865us) server_negotiation.cc:407] Connection header received
0809 04:35:33.433048 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:33.433052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:33.433102 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:33.433219 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:33.433954 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.434697 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.435744 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:33.436080 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:33.436148 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:33.437817 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:33.437821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:33.437823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:33.437851 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:33.439248 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.439650 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.439654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.439656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.439735 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:33.439984 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:33.439987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:33.439989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:33.440172 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:33.440283 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:33.440528 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 04:35:33.440650 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260809 04:35:34.385319 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.384833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:46455)
0809 04:35:34.385098 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.385212 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20260809 04:35:34.385619 30642 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.391017 30743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.390545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:37237)
0809 04:35:34.390802 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.390903 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
W20260809 04:35:34.391299 30428 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.393471 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.393055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:52139)
0809 04:35:34.393296 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.393370 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":62,"threads_started":1}
W20260809 04:35:34.393707 30224 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.897562 30745 raft_consensus.cc:482] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce7fd35486204c12aef2ac2792ad5cec)
I20260809 04:35:34.897792 30745 raft_consensus.cc:504] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } }
I20260809 04:35:34.898166 30746 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce7fd35486204c12aef2ac2792ad5cec)
I20260809 04:35:34.898350 30746 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } }
I20260809 04:35:34.898808 30745 leader_election.cc:288] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937), e1c9cc2ed7e8400186869797c90d58a9 (127.19.92.61:40731)
I20260809 04:35:34.899374 30746 leader_election.cc:288] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937), d649133147ca40869e119ca7e2fe6710 (127.19.92.60:33155)
I20260809 04:35:34.899528 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.898998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52050)
0809 04:35:34.899324 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.899409 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":92,"threads_started":1}
W20260809 04:35:34.899792 29937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.899853 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.899419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52058)
0809 04:35:34.899692 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.899763 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":49,"threads_started":1}
W20260809 04:35:34.900033 29675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.900637 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.900410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52062)
0809 04:35:34.900502 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.900556 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 04:35:34.900758 29937 leader_election.cc:334] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937): Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.901383 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.901125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52070)
0809 04:35:34.901260 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.901313 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260809 04:35:34.901546 29675 leader_election.cc:334] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937): Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.903715 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.899834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.19.92.60:33155)
0809 04:35:34.900042 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:34.900047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:34.900063 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:35:34.900223 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:34.900227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:34.900270 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:34.900344 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:34.900749 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:34.901514 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:34.902655 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:34.903229 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:34.903305 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:34.903356 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:34.903573 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 04:35:34.903628 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260809 04:35:34.903754 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.899721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33155 (local address 127.0.0.1:37120)
0809 04:35:34.899989 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.900003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:34.900097 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:34.900495 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:34.900498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:34.900519 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:34.900622 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:34.900629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:34.901620 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:34.901624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:34.902535 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:34.902544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:34.903322 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:34.903326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:34.903460 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:34.903486 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 04:35:34.903561 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 04:35:34.903631 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":19,"thread_start_us":70,"threads_started":1}
I20260809 04:35:34.904296 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.899124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.19.92.61:40731)
0809 04:35:34.900532 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:34.900536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:34.900550 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:34.900593 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:34.900596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:34.900703 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:34.900789 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:34.901126 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:34.901859 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:34.903107 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:34.903672 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:34.903762 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:34.903814 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:34.904157 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 04:35:34.904215 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":1345,"thread_start_us":65,"threads_started":1}
I20260809 04:35:34.904250 29971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1c9cc2ed7e8400186869797c90d58a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d649133147ca40869e119ca7e2fe6710" is_pre_election: true
I20260809 04:35:34.904376 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.899025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:40731 (local address 127.0.0.1:54940)
0809 04:35:34.899357 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.899372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:34.899481 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:34.900897 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:34.900899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:34.900920 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:34.901019 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:34.901027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:34.902021 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:34.902025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:34.902990 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:34.903001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:34.903958 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:34.903965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:34.904049 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:34.904068 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:35:34.904143 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 04:35:34.904211 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"spinlock_wait_cycles":13568,"thread_start_us":80,"threads_started":1}
I20260809 04:35:34.904507 29971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1c9cc2ed7e8400186869797c90d58a9 in term 1.
I20260809 04:35:34.904803 29705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d649133147ca40869e119ca7e2fe6710" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1c9cc2ed7e8400186869797c90d58a9" is_pre_election: true
I20260809 04:35:34.904856 29671 leader_election.cc:302] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d649133147ca40869e119ca7e2fe6710, e1c9cc2ed7e8400186869797c90d58a9; no voters: ce7fd35486204c12aef2ac2792ad5cec
I20260809 04:35:34.904983 29705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d649133147ca40869e119ca7e2fe6710 in term 1.
I20260809 04:35:34.905066 30746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 04:35:34.905117 30746 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce7fd35486204c12aef2ac2792ad5cec)
I20260809 04:35:34.905182 30746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:35:34.905328 29933 leader_election.cc:302] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d649133147ca40869e119ca7e2fe6710, e1c9cc2ed7e8400186869797c90d58a9; no voters: ce7fd35486204c12aef2ac2792ad5cec
I20260809 04:35:34.905537 30745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 04:35:34.905586 30745 raft_consensus.cc:482] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ce7fd35486204c12aef2ac2792ad5cec)
I20260809 04:35:34.905611 30745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:35:34.906319 30746 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } }
I20260809 04:35:34.906551 30745 raft_consensus.cc:504] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } }
I20260809 04:35:34.906627 30746 leader_election.cc:288] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [CANDIDATE]: Term 2 election: Requested vote from peers ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937), d649133147ca40869e119ca7e2fe6710 (127.19.92.60:33155)
I20260809 04:35:34.906914 30745 leader_election.cc:288] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 2 election: Requested vote from peers ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937), e1c9cc2ed7e8400186869797c90d58a9 (127.19.92.61:40731)
I20260809 04:35:34.906965 29971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1c9cc2ed7e8400186869797c90d58a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d649133147ca40869e119ca7e2fe6710"
I20260809 04:35:34.907024 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.906772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52086)
0809 04:35:34.906894 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.906948 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 04:35:34.907135 29971 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e1c9cc2ed7e8400186869797c90d58a9 in current term 2: Already voted for candidate d649133147ca40869e119ca7e2fe6710 in this term.
I20260809 04:35:34.907444 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.907086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52096)
0809 04:35:34.907332 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.907382 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260809 04:35:34.907610 29705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d649133147ca40869e119ca7e2fe6710" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1c9cc2ed7e8400186869797c90d58a9"
I20260809 04:35:34.907761 29705 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d649133147ca40869e119ca7e2fe6710 in current term 2: Already voted for candidate e1c9cc2ed7e8400186869797c90d58a9 in this term.
I20260809 04:35:34.907992 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.907809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52126)
0809 04:35:34.907899 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.907945 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 04:35:34.908114 29937 leader_election.cc:334] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937): Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.908147 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:34.907476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52112)
0809 04:35:34.908034 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:34.908086 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493}
I20260809 04:35:34.908177 29937 leader_election.cc:302] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d649133147ca40869e119ca7e2fe6710; no voters: ce7fd35486204c12aef2ac2792ad5cec, e1c9cc2ed7e8400186869797c90d58a9
W20260809 04:35:34.908260 29675 leader_election.cc:334] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937): Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:34.908313 30745 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 04:35:34.908339 29675 leader_election.cc:302] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1c9cc2ed7e8400186869797c90d58a9; no voters: ce7fd35486204c12aef2ac2792ad5cec, d649133147ca40869e119ca7e2fe6710
I20260809 04:35:34.908522 30746 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 04:35:35.386740 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:35.386253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:57799)
0809 04:35:35.386558 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:35.386629 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260809 04:35:35.392196 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:35.391800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:54447)
0809 04:35:35.392060 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:35.392116 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260809 04:35:35.394654 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:35.394233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:49167)
0809 04:35:35.394512 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:35.394574 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260809 04:35:36.387887 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.387478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:59425)
0809 04:35:36.387727 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.387788 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260809 04:35:36.393303 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.392820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:57147)
0809 04:35:36.393121 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.393212 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":74,"threads_started":1}
I20260809 04:35:36.395565 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.395182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:53209)
0809 04:35:36.395417 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.395477 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260809 04:35:36.532696 30774 raft_consensus.cc:482] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:36.532840 30774 raft_consensus.cc:504] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } }
I20260809 04:35:36.533349 30774 leader_election.cc:288] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937), e1c9cc2ed7e8400186869797c90d58a9 (127.19.92.61:40731)
I20260809 04:35:36.533800 29705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d649133147ca40869e119ca7e2fe6710" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1c9cc2ed7e8400186869797c90d58a9" is_pre_election: true
I20260809 04:35:36.533916 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.533543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52338)
0809 04:35:36.533769 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.533827 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260809 04:35:36.533950 29705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d649133147ca40869e119ca7e2fe6710 in term 2.
I20260809 04:35:36.534272 29933 leader_election.cc:302] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d649133147ca40869e119ca7e2fe6710, e1c9cc2ed7e8400186869797c90d58a9; no voters:
I20260809 04:35:36.534425 30774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 04:35:36.534480 30774 raft_consensus.cc:482] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:36.534518 30774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 2 FOLLOWER]: Advancing to term 3
I20260809 04:35:36.534742 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.534502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52348)
0809 04:35:36.534647 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.534687 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260809 04:35:36.534854 29937 leader_election.cc:334] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937): Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:36.535612 30774 raft_consensus.cc:504] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } }
I20260809 04:35:36.535904 30774 leader_election.cc:288] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 3 election: Requested vote from peers ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937), e1c9cc2ed7e8400186869797c90d58a9 (127.19.92.61:40731)
I20260809 04:35:36.536212 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.536029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52356)
0809 04:35:36.536119 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.536161 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 04:35:36.536248 29705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d649133147ca40869e119ca7e2fe6710" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1c9cc2ed7e8400186869797c90d58a9"
I20260809 04:35:36.536374 29705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 04:35:36.536831 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.536663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52358)
0809 04:35:36.536744 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.536784 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 04:35:36.536926 29937 leader_election.cc:334] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937): Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
I20260809 04:35:36.537493 29705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d649133147ca40869e119ca7e2fe6710 in term 3.
I20260809 04:35:36.537758 29933 leader_election.cc:302] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d649133147ca40869e119ca7e2fe6710, e1c9cc2ed7e8400186869797c90d58a9; no voters: ce7fd35486204c12aef2ac2792ad5cec
I20260809 04:35:36.537927 30774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 04:35:36.538131 30774 raft_consensus.cc:686] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [term 3 LEADER]: Becoming Leader. State: Replica: d649133147ca40869e119ca7e2fe6710, State: Running, Role: LEADER
I20260809 04:35:36.538380 30774 consensus_queue.cc:237] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } }
I20260809 04:35:36.539774 30777 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d649133147ca40869e119ca7e2fe6710. Latest consensus state: current_term: 3 leader_uuid: "d649133147ca40869e119ca7e2fe6710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:36.539889 30777 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:36.540169 30780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:36.542418 30780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:36.542811 30780 catalog_manager.cc:1259] Loaded cluster ID: 1420b5688c154c7eb3506004a0575921
I20260809 04:35:36.542855 30780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:36.543401 30780 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:36.543771 30780 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Loaded TSK: 0
I20260809 04:35:36.544450 30780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:35:36.562444 29945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:36.565596 29945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:36.568583 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.568328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52384)
0809 04:35:36.568442 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.568506 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 04:35:36.568707 29705 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [term 3 FOLLOWER]: Refusing update from remote peer d649133147ca40869e119ca7e2fe6710: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260809 04:35:36.569286 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.569011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.0.0.1:52396)
0809 04:35:36.569120 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.569199 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 04:35:36.569211 30777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 04:35:36.569407 29937 consensus_peers.cc:487] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 -> Peer ce7fd35486204c12aef2ac2792ad5cec (127.19.92.62:34937): Couldn't send request to peer ce7fd35486204c12aef2ac2792ad5cec. Status: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 04:35:36.570711 30784 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d649133147ca40869e119ca7e2fe6710. Latest consensus state: current_term: 3 leader_uuid: "d649133147ca40869e119ca7e2fe6710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:36.570833 30784 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:35:36.571960 30777 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d649133147ca40869e119ca7e2fe6710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:36.572090 30777 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:36.572351 30784 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d649133147ca40869e119ca7e2fe6710" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce7fd35486204c12aef2ac2792ad5cec" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34937 } } peers { permanent_uuid: "e1c9cc2ed7e8400186869797c90d58a9" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 40731 } attrs { promote: false } } peers { permanent_uuid: "d649133147ca40869e119ca7e2fe6710" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33155 } attrs { promote: false } } }
I20260809 04:35:36.572520 30784 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9cc2ed7e8400186869797c90d58a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:35:36.582597 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.578104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45189 (local address 127.0.0.1:46952)
0809 04:35:36.578258 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.578274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:36.578379 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:36.578778 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:36.578781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:36.578802 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:36.578927 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.578934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.580014 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.580017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.581259 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.581269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.582215 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.582218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.582301 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.582331 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 04:35:36.582379 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 04:35:36.582452 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":48}
I20260809 04:35:36.583221 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.578242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.19.92.1:45189)
0809 04:35:36.578525 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:36.578530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:36.578544 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:36.578596 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:36.578599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:36.578649 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:36.579083 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:36.579204 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.579876 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.581383 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.582079 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.582871 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.582974 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:36.583016 (+ 42us) server_negotiation.cc:299] Negotiation successful
0809 04:35:36.583092 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260809 04:35:36.583565 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.578743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.19.92.2:45401)
0809 04:35:36.578988 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:36.578993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:36.579545 (+ 552us) server_negotiation.cc:407] Connection header received
0809 04:35:36.579609 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:36.579612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:36.579660 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:36.579739 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:36.580075 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.580879 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.582182 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.582719 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.582820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.582884 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:36.583169 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 04:35:36.583240 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260809 04:35:36.583345 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.578362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:45401 (local address 127.0.0.1:51512)
0809 04:35:36.579420 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.579433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:36.579523 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:36.579848 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:36.579851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:36.579869 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:36.579955 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.579962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.581006 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.581009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.582070 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.582081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.582943 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.582947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.583052 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.583071 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:35:36.583137 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 04:35:36.583207 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":115,"threads_started":1}
I20260809 04:35:36.583797 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.578808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:45393 (local address 127.0.0.1:42436)
0809 04:35:36.579649 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.579658 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:36.579726 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:36.579933 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:36.579936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:36.579951 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:36.580045 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.580050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.580784 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.580787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.582117 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.582127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.583429 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.583433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.583523 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.583541 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 04:35:36.583612 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 04:35:36.583676 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260809 04:35:36.583875 30128 tablet_service.cc:1467] Processing CreateTablet for tablet b60f2b1f00ee4a5b8731239db8db7b21 (DEFAULT_TABLE table=test-table [id=28137b0441a44dc691507d4d53838295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:36.584216 30128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60f2b1f00ee4a5b8731239db8db7b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:36.584326 30335 tablet_service.cc:1467] Processing CreateTablet for tablet b60f2b1f00ee4a5b8731239db8db7b21 (DEFAULT_TABLE table=test-table [id=28137b0441a44dc691507d4d53838295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:36.584606 30335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60f2b1f00ee4a5b8731239db8db7b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:36.585575 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.579165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.19.92.3:45393)
0809 04:35:36.579389 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:36.579394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:36.579730 (+ 336us) server_negotiation.cc:407] Connection header received
0809 04:35:36.579797 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:36.579800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:36.579847 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:36.579922 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:36.580142 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.580681 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.582249 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.583175 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.583284 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.583355 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:36.585367 (+ 2012us) server_negotiation.cc:299] Negotiation successful
0809 04:35:36.585465 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260809 04:35:36.586074 30542 tablet_service.cc:1467] Processing CreateTablet for tablet b60f2b1f00ee4a5b8731239db8db7b21 (DEFAULT_TABLE table=test-table [id=28137b0441a44dc691507d4d53838295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:36.586324 30542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60f2b1f00ee4a5b8731239db8db7b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:36.587191 30805 tablet_bootstrap.cc:492] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf: Bootstrap starting.
I20260809 04:35:36.588146 30805 tablet_bootstrap.cc:654] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:36.588725 30806 tablet_bootstrap.cc:492] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: Bootstrap starting.
I20260809 04:35:36.589063 30804 tablet_bootstrap.cc:492] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149: Bootstrap starting.
I20260809 04:35:36.589728 30806 tablet_bootstrap.cc:654] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:36.590018 30804 tablet_bootstrap.cc:654] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:36.590103 30805 tablet_bootstrap.cc:492] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf: No bootstrap required, opened a new log
I20260809 04:35:36.590200 30805 ts_tablet_manager.cc:1397] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:36.590580 30805 raft_consensus.cc:348] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.590703 30805 raft_consensus.cc:374] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:36.590749 30805 raft_consensus.cc:729] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404364fda6b846dcb4dedb42b23e7caf, State: Initialized, Role: FOLLOWER
I20260809 04:35:36.590888 30805 consensus_queue.cc:260] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.590960 30806 tablet_bootstrap.cc:492] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: No bootstrap required, opened a new log
I20260809 04:35:36.591024 30806 ts_tablet_manager.cc:1397] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:36.591262 30805 ts_tablet_manager.cc:1428] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:36.591307 30804 tablet_bootstrap.cc:492] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149: No bootstrap required, opened a new log
I20260809 04:35:36.591416 30806 raft_consensus.cc:348] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.591517 30806 raft_consensus.cc:374] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:36.591547 30806 raft_consensus.cc:729] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e157049549f428d8798db9612b2b72c, State: Initialized, Role: FOLLOWER
I20260809 04:35:36.591658 30806 consensus_queue.cc:260] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.592017 30806 ts_tablet_manager.cc:1428] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:36.592027 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.591668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:35301)
0809 04:35:36.591901 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.591958 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260809 04:35:36.593271 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.592914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:47181)
0809 04:35:36.593110 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.593200 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260809 04:35:36.593827 30427 heartbeater.cc:503] Master 127.19.92.60:33155 was elected leader, sending a full tablet report...
I20260809 04:35:36.594432 30639 heartbeater.cc:503] Master 127.19.92.60:33155 was elected leader, sending a full tablet report...
I20260809 04:35:36.594938 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.594715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:33977)
0809 04:35:36.594804 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.594855 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260809 04:35:36.595259 30804 ts_tablet_manager.cc:1397] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260809 04:35:36.595329 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.595102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:47615)
0809 04:35:36.595216 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.595266 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 04:35:36.595707 30804 raft_consensus.cc:348] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.595829 30804 raft_consensus.cc:374] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:36.595862 30804 raft_consensus.cc:729] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4c93a7abd24a6492b3d1eb45c25149, State: Initialized, Role: FOLLOWER
I20260809 04:35:36.595985 30804 consensus_queue.cc:260] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.596474 30804 ts_tablet_manager.cc:1428] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:36.596800 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.596606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:47769)
0809 04:35:36.596691 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.596742 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260809 04:35:36.597501 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.597239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:35633)
0809 04:35:36.597382 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.597432 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 04:35:36.597687 30222 heartbeater.cc:503] Master 127.19.92.60:33155 was elected leader, sending a full tablet report...
I20260809 04:35:36.698884 30813 raft_consensus.cc:482] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:36.699059 30813 raft_consensus.cc:504] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.700194 30813 leader_election.cc:288] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e4c93a7abd24a6492b3d1eb45c25149 (127.19.92.1:45189), 404364fda6b846dcb4dedb42b23e7caf (127.19.92.2:45401)
I20260809 04:35:36.705081 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.700839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:57929 (local address 127.19.92.2:45401)
0809 04:35:36.700964 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:36.700968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:36.701192 (+ 224us) server_negotiation.cc:407] Connection header received
0809 04:35:36.701329 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:36.701332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:36.701384 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:36.701461 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:36.701755 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.702402 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.703724 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.704329 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.704456 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.704524 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:36.704847 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 04:35:36.704936 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 04:35:36.705076 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.700546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:49809 (local address 127.19.92.1:45189)
0809 04:35:36.700701 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:36.700706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:36.700725 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:35:36.700795 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:36.700798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:36.700852 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:36.700929 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:36.701360 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.702106 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.703668 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.704318 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.704456 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.704544 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:36.704847 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 04:35:36.704937 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:35:36.705078 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.700693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:45401 (local address 127.19.92.3:57929)
0809 04:35:36.701074 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.701086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:36.701236 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:36.701525 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:36.701529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:36.701550 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:36.701649 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.701654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.702504 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.702507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.703591 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.703599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.704509 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.704514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.704659 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.704701 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 04:35:36.704783 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 04:35:36.704897 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":34,"thread_start_us":144,"threads_started":1}
I20260809 04:35:36.705077 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.700416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45189 (local address 127.19.92.3:49809)
0809 04:35:36.700563 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.700581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:36.700700 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:36.701043 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:36.701046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:36.701069 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:36.701255 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.701262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.702206 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.702209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.703508 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:36.703520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.704509 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.704514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:36.704665 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.704705 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 04:35:36.704783 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:35:36.704897 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":73}
I20260809 04:35:36.705708 30363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60f2b1f00ee4a5b8731239db8db7b21" candidate_uuid: "4e157049549f428d8798db9612b2b72c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404364fda6b846dcb4dedb42b23e7caf" is_pre_election: true
I20260809 04:35:36.705761 30151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60f2b1f00ee4a5b8731239db8db7b21" candidate_uuid: "4e157049549f428d8798db9612b2b72c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" is_pre_election: true
I20260809 04:35:36.705899 30363 raft_consensus.cc:2454] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e157049549f428d8798db9612b2b72c in term 0.
I20260809 04:35:36.705989 30151 raft_consensus.cc:2454] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e157049549f428d8798db9612b2b72c in term 0.
I20260809 04:35:36.706303 30465 leader_election.cc:302] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 404364fda6b846dcb4dedb42b23e7caf, 4e157049549f428d8798db9612b2b72c; no voters:
I20260809 04:35:36.706598 30813 raft_consensus.cc:2781] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:36.706661 30813 raft_consensus.cc:482] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:36.706700 30813 raft_consensus.cc:3037] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:36.707710 30813 raft_consensus.cc:504] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.708058 30813 leader_election.cc:288] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [CANDIDATE]: Term 1 election: Requested vote from peers 2e4c93a7abd24a6492b3d1eb45c25149 (127.19.92.1:45189), 404364fda6b846dcb4dedb42b23e7caf (127.19.92.2:45401)
I20260809 04:35:36.708405 30151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60f2b1f00ee4a5b8731239db8db7b21" candidate_uuid: "4e157049549f428d8798db9612b2b72c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4c93a7abd24a6492b3d1eb45c25149"
I20260809 04:35:36.708495 30363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60f2b1f00ee4a5b8731239db8db7b21" candidate_uuid: "4e157049549f428d8798db9612b2b72c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404364fda6b846dcb4dedb42b23e7caf"
I20260809 04:35:36.708540 30151 raft_consensus.cc:3037] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:36.708622 30363 raft_consensus.cc:3037] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:36.709730 30363 raft_consensus.cc:2454] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e157049549f428d8798db9612b2b72c in term 1.
I20260809 04:35:36.709730 30151 raft_consensus.cc:2454] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e157049549f428d8798db9612b2b72c in term 1.
I20260809 04:35:36.710040 30465 leader_election.cc:302] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4c93a7abd24a6492b3d1eb45c25149, 4e157049549f428d8798db9612b2b72c; no voters:
I20260809 04:35:36.710248 30813 raft_consensus.cc:2781] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:36.710465 30813 raft_consensus.cc:686] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 1 LEADER]: Becoming Leader. State: Replica: 4e157049549f428d8798db9612b2b72c, State: Running, Role: LEADER
I20260809 04:35:36.710940 30813 consensus_queue.cc:237] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } }
I20260809 04:35:36.711187 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.710943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:45553)
0809 04:35:36.711056 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.711111 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 04:35:36.713557 29945 catalog_manager.cc:5495] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e157049549f428d8798db9612b2b72c (127.19.92.3). New cstate: current_term: 1 leader_uuid: "4e157049549f428d8798db9612b2b72c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e157049549f428d8798db9612b2b72c" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 45393 } health_report { overall_health: HEALTHY } } }
W20260809 04:35:36.727924 30644 tablet.cc:2369] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:35:36.736690 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.733483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.19.92.3:45393)
0809 04:35:36.733608 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:36.733612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:36.733624 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:35:36.733727 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:36.733729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:36.733785 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:36.733876 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:36.734073 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.734598 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.735386 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:36.735607 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:36.735678 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:36.735868 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:36.736335 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:36.736421 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:36.736502 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 04:35:36.736572 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 04:35:36.739677 30363 raft_consensus.cc:1261] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 1 FOLLOWER]: Refusing update from remote peer 4e157049549f428d8798db9612b2b72c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:36.739869 30151 raft_consensus.cc:1261] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 FOLLOWER]: Refusing update from remote peer 4e157049549f428d8798db9612b2b72c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:36.740260 30816 consensus_queue.cc:1035] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [LEADER]: Connected to new peer: Peer: permanent_uuid: "404364fda6b846dcb4dedb42b23e7caf" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:36.740422 30813 consensus_queue.cc:1035] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4c93a7abd24a6492b3d1eb45c25149" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:36.742281 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.741760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:34859)
0809 04:35:36.742129 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.742198 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291}
I20260809 04:35:36.742859 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.742575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:53133)
0809 04:35:36.742731 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.742786 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 04:35:36.745985 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.745549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:37875)
0809 04:35:36.745835 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.745901 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260809 04:35:36.747395 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.747130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.1:41293)
0809 04:35:36.747272 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.747325 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260809 04:35:36.748934 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.748658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.3:53679)
0809 04:35:36.748800 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.748850 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 04:35:36.750178 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.749876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34937 (local address 127.19.92.2:54203)
0809 04:35:36.750063 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.750111 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260809 04:35:36.755488 29945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:36.755682 29945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:36.759840 29945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d649133147ca40869e119ca7e2fe6710: Sending DeleteTablet for 3 replicas of tablet b60f2b1f00ee4a5b8731239db8db7b21
I20260809 04:35:36.760655 30128 tablet_service.cc:1514] Processing DeleteTablet for tablet b60f2b1f00ee4a5b8731239db8db7b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:36 UTC) from {username='oryx', principal='oryx/127.19.92.60@KRBTEST.COM'} at 127.0.0.1:46952
I20260809 04:35:36.760757 30335 tablet_service.cc:1514] Processing DeleteTablet for tablet b60f2b1f00ee4a5b8731239db8db7b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:36 UTC) from {username='oryx', principal='oryx/127.19.92.60@KRBTEST.COM'} at 127.0.0.1:51512
I20260809 04:35:36.760937 30542 tablet_service.cc:1514] Processing DeleteTablet for tablet b60f2b1f00ee4a5b8731239db8db7b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:36 UTC) from {username='oryx', principal='oryx/127.19.92.60@KRBTEST.COM'} at 127.0.0.1:42436
I20260809 04:35:36.761475 30836 tablet_replica.cc:330] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149: stopping tablet replica
I20260809 04:35:36.761668 30836 raft_consensus.cc:2229] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:36.761819 30836 raft_consensus.cc:2258] T b60f2b1f00ee4a5b8731239db8db7b21 P 2e4c93a7abd24a6492b3d1eb45c25149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:36.762036 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 29888
I20260809 04:35:36.765028 30840 tablet_replica.cc:330] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: stopping tablet replica
I20260809 04:35:36.765254 30840 raft_consensus.cc:2229] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:36.765715 30840 raft_consensus.cc:2258] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:36.770077 30840 ts_tablet_manager.cc:1905] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:36.773231 30840 ts_tablet_manager.cc:1918] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:36.773375 30840 log.cc:1196] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/b60f2b1f00ee4a5b8731239db8db7b21
I20260809 04:35:36.773651 30839 tablet_replica.cc:330] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf: stopping tablet replica
I20260809 04:35:36.773674 30840 ts_tablet_manager.cc:1939] T b60f2b1f00ee4a5b8731239db8db7b21 P 4e157049549f428d8798db9612b2b72c: Deleting consensus metadata
I20260809 04:35:36.773825 30839 raft_consensus.cc:2229] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:36.774008 30839 raft_consensus.cc:2258] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:36.774505 29937 catalog_manager.cc:4841] TS 4e157049549f428d8798db9612b2b72c (127.19.92.3:45393): tablet b60f2b1f00ee4a5b8731239db8db7b21 (table test-table [id=28137b0441a44dc691507d4d53838295]) successfully deleted
I20260809 04:35:36.781788 30839 ts_tablet_manager.cc:1905] T b60f2b1f00ee4a5b8731239db8db7b21 P 404364fda6b846dcb4dedb42b23e7caf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 04:35:36.784005 30796 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:36.783583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45189 (local address 127.0.0.1:46956)
0809 04:35:36.783803 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.783862 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:45189: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":132}
I20260809 04:35:36.784099 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 30236
W20260809 04:35:36.787288 29934 catalog_manager.cc:4584] TS 2e4c93a7abd24a6492b3d1eb45c25149 (127.19.92.1:45189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b60f2b1f00ee4a5b8731239db8db7b21: Network error: Client connection negotiation failed: client connection to 127.19.92.1:45189: connect: Connection reset by peer (error 104)
I20260809 04:35:36.796573 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 30437
I20260809 04:35:36.796916 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:36.796593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:45401 (local address 127.0.0.1:51528)
0809 04:35:36.796755 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:36.796818 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.2:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 04:35:36.808192 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 29486
I20260809 04:35:36.821583 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 29638
2026-08-09T04:35:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7095 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:36 dist-test-slave-rmhg krb5kdc[30843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:36 dist-test-slave-rmhg krb5kdc[30843](info): set up 2 sockets
Aug 09 04:35:36 dist-test-slave-rmhg krb5kdc[30843](info): commencing operation
krb5kdc: starting...
W20260809 04:35:37.963631 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:38 dist-test-slave-rmhg krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.62.keytab.
Entry for principal oryx/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.62.keytab.
I20260809 04:35:38.092289 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:35441
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.62.keytab
--principal=oryx/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:35441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:38.208475 30861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:38.208782 30861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:38.208843 30861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:38.213027 30861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:38.213107 30861 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 04:35:38.213131 30861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:38.213176 30861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:38.213202 30861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:38.213232 30861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:38.218858 30861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:35441
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.62.keytab
--principal=oryx/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:35441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:38.220041 30861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:38.221122 30861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:38.229806 30875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:38.229828 30868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:38.229828 30870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:38.230202 30861 server_base.cc:1034] running on GCE node
I20260809 04:35:38.230743 30861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:38.231859 30861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:38.233033 30861 hybrid_clock.cc:648] HybridClock initialized: now 1786250138233013 us; error 34 us; skew 500 ppm
Aug 09 04:35:38 dist-test-slave-rmhg krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:38.235862 30861 init.cc:377] Logged in from keytab as oryx/127.19.92.62@KRBTEST.COM (short username oryx)
I20260809 04:35:38.236778 30861 webserver.cc:460] Webserver started at http://127.19.92.62:45235/ using document root <none> and password file <none>
I20260809 04:35:38.237380 30861 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:38.237449 30861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:38.237639 30861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:38.239886 30861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "aa8442cfa7de411ea293f57331d8091a"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "b36d74e645c0cc13f4354cb6766b9946"
server_key_iv: "f0222be74afb68e9109b5c9701388959"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.240511 30861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "aa8442cfa7de411ea293f57331d8091a"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "b36d74e645c0cc13f4354cb6766b9946"
server_key_iv: "f0222be74afb68e9109b5c9701388959"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.244695 30861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:35:38.247305 30883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:38.248715 30861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:38.248839 30861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "aa8442cfa7de411ea293f57331d8091a"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "b36d74e645c0cc13f4354cb6766b9946"
server_key_iv: "f0222be74afb68e9109b5c9701388959"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.248935 30861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:38.264274 30861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:38.267796 30861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:38.267970 30861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:38.275974 30861 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:35441
I20260809 04:35:38.276006 30976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:35441 every 8 connection(s)
I20260809 04:35:38.276993 30861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:38.280113 30978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:38.280208 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 30861
I20260809 04:35:38.280350 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:38.280637 19824 external_mini_cluster.cc:1428] Setting key 99475ecc6feae639de1f669c5c41b36c
I20260809 04:35:38.287789 30978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: Bootstrap starting.
I20260809 04:35:38.290475 30978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: Neither blocks nor log segments found. Creating new log.
Aug 09 04:35:38 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:38.291361 30978 log.cc:824] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:38.293959 30978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: No bootstrap required, opened a new log
I20260809 04:35:38.296906 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.282201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.19.92.62:35441)
0809 04:35:38.282668 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:38.282694 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:38.282742 (+ 48us) server_negotiation.cc:407] Connection header received
0809 04:35:38.283696 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:38.283724 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:38.284192 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:38.284613 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:38.285364 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.286900 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.287795 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.288510 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.288667 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.291114 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:38.291121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:38.291140 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:38.291178 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:38.293752 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.294219 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.294226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.294233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.294389 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.294628 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.294631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.294633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.295186 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:38.295379 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:38.295579 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 04:35:38.295896 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":130,"threads_started":1}
I20260809 04:35:38.297907 30978 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8442cfa7de411ea293f57331d8091a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35441 } }
I20260809 04:35:38.298211 30978 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:38.298275 30978 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8442cfa7de411ea293f57331d8091a, State: Initialized, Role: FOLLOWER
I20260809 04:35:38.298770 30978 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8442cfa7de411ea293f57331d8091a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35441 } }
I20260809 04:35:38.298942 30978 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:38.299010 30978 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:38.299109 30978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:38.300062 30978 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8442cfa7de411ea293f57331d8091a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35441 } }
I20260809 04:35:38.300405 30978 leader_election.cc:302] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8442cfa7de411ea293f57331d8091a; no voters:
I20260809 04:35:38.300719 30978 leader_election.cc:288] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:38.300835 30986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:38.301035 30986 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [term 1 LEADER]: Becoming Leader. State: Replica: aa8442cfa7de411ea293f57331d8091a, State: Running, Role: LEADER
I20260809 04:35:38.301425 30986 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8442cfa7de411ea293f57331d8091a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35441 } }
I20260809 04:35:38.302006 30978 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:38.303038 30988 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8442cfa7de411ea293f57331d8091a. Latest consensus state: current_term: 1 leader_uuid: "aa8442cfa7de411ea293f57331d8091a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8442cfa7de411ea293f57331d8091a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35441 } } }
I20260809 04:35:38.303261 30988 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:38.303411 30987 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa8442cfa7de411ea293f57331d8091a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8442cfa7de411ea293f57331d8091a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35441 } } }
I20260809 04:35:38.303496 30987 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:38.303809 31000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:38.306937 31000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:38.313275 31000 catalog_manager.cc:1347] Generated new cluster ID: 7100357d57204b54b86870ce715a0f77
I20260809 04:35:38.313436 31000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:38.327979 31000 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:38.329126 31000 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:38.337585 31000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: Generated new TSK 0
I20260809 04:35:38.338469 31000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab.
Entry for principal oryx/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab.
I20260809 04:35:38.388584 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35441
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab
--principal=oryx/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:38.507980 31018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:38.508273 31018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:38.508320 31018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:38.512475 31018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:38.512547 31018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:38.512585 31018 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:38.512616 31018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:38.518026 31018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab
--principal=oryx/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:35441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:38.519192 31018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:38.520612 31018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:38.530721 31030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:38.530745 31032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:38.530890 31018 server_base.cc:1034] running on GCE node
W20260809 04:35:38.530718 31035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:38.531497 31018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:38.532202 31018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:38.533376 31018 hybrid_clock.cc:648] HybridClock initialized: now 1786250138533357 us; error 34 us; skew 500 ppm
Aug 09 04:35:38 dist-test-slave-rmhg krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:38.536232 31018 init.cc:377] Logged in from keytab as oryx/127.19.92.1@KRBTEST.COM (short username oryx)
I20260809 04:35:38.537225 31018 webserver.cc:460] Webserver started at http://127.19.92.1:34039/ using document root <none> and password file <none>
I20260809 04:35:38.537788 31018 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:38.537858 31018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:38.538079 31018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:38.540208 31018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "b5d9f70c57e9f0a0291d4eb99219f25e"
server_key_iv: "2faeb8a06d1cfcc7e04c50ff2447e752"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.540760 31018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "b5d9f70c57e9f0a0291d4eb99219f25e"
server_key_iv: "2faeb8a06d1cfcc7e04c50ff2447e752"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.544513 31018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:35:38.547096 31042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:38.548552 31018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:38.548666 31018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "b5d9f70c57e9f0a0291d4eb99219f25e"
server_key_iv: "2faeb8a06d1cfcc7e04c50ff2447e752"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.548764 31018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:38.568727 31018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:38.571955 31018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:38.572144 31018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:38.573807 31018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:38.573863 31018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:38.573916 31018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:38.573943 31018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:35:38 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:38.584666 31018 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:38877
I20260809 04:35:38.584740 31218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:38877 every 8 connection(s)
I20260809 04:35:38.585845 31018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:38.587815 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 31018
I20260809 04:35:38.587947 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:38.588163 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.574653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:35519 (local address 127.19.92.62:35441)
0809 04:35:38.574811 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:38.574816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:38.575719 (+ 903us) server_negotiation.cc:407] Connection header received
0809 04:35:38.576982 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:38.576987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:38.577061 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:38.577122 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:38.579116 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.579730 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.580992 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.581312 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.581414 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.584110 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:38.584114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:38.584116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:38.584146 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:38.585909 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.586555 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.586558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.586560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.586659 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.586989 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.586992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.586993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.587196 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:38.587271 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:38.587831 (+ 560us) server_negotiation.cc:299] Negotiation successful
0809 04:35:38.587975 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:35:38.588253 19824 external_mini_cluster.cc:1428] Setting key 9ff3dd267dc3da8a03376493b833d874
I20260809 04:35:38.588994 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.575042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35441 (local address 127.19.92.1:35519)
0809 04:35:38.575529 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:38.575590 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:38.576688 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:38.577658 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:38.577668 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:38.578166 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:38.578913 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:38.578938 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.579873 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.579875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:38.580849 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:38.580860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.581500 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.581505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:38.581674 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.582268 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:38.582285 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:38.583909 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:38.586048 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:38.586054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:38.586066 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:38.586395 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:38.586761 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:38.586764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:38.586765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:38.586898 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:38.587298 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:38.587303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:38.587592 (+ 289us) client_negotiation.cc:769] Sending connection context
0809 04:35:38.587762 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 04:35:38.588093 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260809 04:35:38.590322 31220 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35441
I20260809 04:35:38.590472 31220 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:38.590978 31220 heartbeater.cc:511] Master 127.19.92.62:35441 requested a full tablet report, sending...
I20260809 04:35:38.592613 30907 ts_manager.cc:194] Registered new tserver with Master: 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877)
I20260809 04:35:38.594146 30907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:35519
I20260809 04:35:38.600126 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.592605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35441 (local address 127.19.92.1:54499)
0809 04:35:38.592889 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:38.592902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:38.593040 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:38.593297 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:38.593300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:38.593505 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:38.593609 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:38.593615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.594380 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.594383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:38.595062 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:38.595074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.595627 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.595631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:38.595724 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.596291 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:38.596305 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:38.596731 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:38.598587 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:38.598589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:38.598591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:38.598798 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:38.599092 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:38.599094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:38.599096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:38.599179 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:38.599629 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:38.599632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:38.599721 (+ 89us) client_negotiation.cc:769] Sending connection context
0809 04:35:38.599813 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 04:35:38.599937 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260809 04:35:38.600188 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.592667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:54499 (local address 127.19.92.62:35441)
0809 04:35:38.592895 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:38.592899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:38.592968 (+ 69us) server_negotiation.cc:407] Connection header received
0809 04:35:38.593122 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:38.593125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:38.593204 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:38.593281 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:38.593719 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.594236 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.595185 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.595492 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.595604 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.596878 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:38.596882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:38.596885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:38.596912 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:38.598454 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.598896 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.598900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.598902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.598994 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.599262 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.599266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.599267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.599498 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:38.599620 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:38.599900 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 04:35:38.600035 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
WARNING: no policy specified for oryx/127.19.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab.
Entry for principal oryx/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab.
I20260809 04:35:38.620085 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35441
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab
--principal=oryx/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:38.734887 31233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:38.735189 31233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:38.735258 31233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:38.739346 31233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:38.739418 31233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:38.739473 31233 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:38.739530 31233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:38.744937 31233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab
--principal=oryx/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:35441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:38.746078 31233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:38.747145 31233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:38.757040 31243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:38.757205 31233 server_base.cc:1034] running on GCE node
W20260809 04:35:38.757277 31239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:38.757041 31245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:38.757748 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:38.758407 31233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:38.759589 31233 hybrid_clock.cc:648] HybridClock initialized: now 1786250138759565 us; error 37 us; skew 500 ppm
Aug 09 04:35:38 dist-test-slave-rmhg krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:38.762498 31233 init.cc:377] Logged in from keytab as oryx/127.19.92.2@KRBTEST.COM (short username oryx)
I20260809 04:35:38.763504 31233 webserver.cc:460] Webserver started at http://127.19.92.2:34713/ using document root <none> and password file <none>
I20260809 04:35:38.764091 31233 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:38.764161 31233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:38.764444 31233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:38.766619 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "4f5dec8009ca4c1dbac87be3221063c0"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "51dbfb6f20aaef8702482533a5181967"
server_key_iv: "098850676868d91faabcca73918f76f3"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.767184 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "4f5dec8009ca4c1dbac87be3221063c0"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "51dbfb6f20aaef8702482533a5181967"
server_key_iv: "098850676868d91faabcca73918f76f3"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.771042 31233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 04:35:38.773466 31256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:38.774773 31233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:35:38.774904 31233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "4f5dec8009ca4c1dbac87be3221063c0"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "51dbfb6f20aaef8702482533a5181967"
server_key_iv: "098850676868d91faabcca73918f76f3"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.775025 31233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:38.795212 31233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:38.798529 31233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:38.798720 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:38.800300 31233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:38.800361 31233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:38.800418 31233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:38.800447 31233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:38.810248 31233 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:33057
I20260809 04:35:38.810312 31430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:33057 every 8 connection(s)
Aug 09 04:35:38 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:38.811188 31233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:38.814713 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.801131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:34827 (local address 127.19.92.62:35441)
0809 04:35:38.801327 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:38.801332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:38.802250 (+ 918us) server_negotiation.cc:407] Connection header received
0809 04:35:38.803557 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:38.803561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:38.803627 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:38.803728 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:38.805539 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.806381 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.808003 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.808351 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.808486 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.811293 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:38.811297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:38.811300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:38.811332 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:38.812863 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.813389 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.813392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.813393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.813518 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.813778 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.813780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.813781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.813965 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:38.814044 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:38.814489 (+ 445us) server_negotiation.cc:299] Negotiation successful
0809 04:35:38.814594 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 04:35:38.815542 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.801592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35441 (local address 127.19.92.2:34827)
0809 04:35:38.802075 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:38.802129 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:38.803288 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:38.804044 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:38.804052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:38.804542 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:38.805321 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:38.805344 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.806522 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.806527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:38.807875 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:38.807886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.808551 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.808556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:38.808773 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.809444 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:38.809461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:38.811111 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:38.812972 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:38.812976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:38.812989 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:38.813298 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:38.813584 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:38.813586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:38.813588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:38.813713 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:38.814037 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:38.814041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:38.814270 (+ 229us) client_negotiation.cc:769] Sending connection context
0809 04:35:38.814411 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 04:35:38.814701 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260809 04:35:38.816684 31432 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35441
I20260809 04:35:38.816826 31432 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:38.817423 31432 heartbeater.cc:511] Master 127.19.92.62:35441 requested a full tablet report, sending...
I20260809 04:35:38.817898 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 31233
I20260809 04:35:38.818011 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:38.818341 19824 external_mini_cluster.cc:1428] Setting key 7bf1d1450a80c5ad28620f198f32334d
I20260809 04:35:38.818562 30909 ts_manager.cc:194] Registered new tserver with Master: 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057)
I20260809 04:35:38.819312 30909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:34827
I20260809 04:35:38.826164 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.819082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:58491 (local address 127.19.92.62:35441)
0809 04:35:38.819311 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:38.819315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:38.819376 (+ 61us) server_negotiation.cc:407] Connection header received
0809 04:35:38.819544 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:38.819546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:38.819590 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:38.819661 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:38.820191 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.820852 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.821797 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.822046 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.822120 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.823264 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:38.823267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:38.823269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:38.823290 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:38.824587 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.825041 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.825043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.825044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.825113 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:38.825381 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:38.825383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:38.825384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:38.825595 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:38.825671 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:38.825947 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 04:35:38.826051 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260809 04:35:38.826236 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:38.819043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35441 (local address 127.19.92.2:58491)
0809 04:35:38.819296 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:38.819310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:38.819456 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:38.819705 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:38.819708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:38.819921 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:38.820061 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:38.820069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.820997 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.821000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:38.821654 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:38.821666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:38.822143 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:38.822147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:38.822239 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:38.822694 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:38.822708 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:38.823122 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:38.824699 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:38.824702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:38.824704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:38.824948 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:38.825202 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:38.825205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:38.825207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:38.825300 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:38.825688 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:38.825692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:38.825795 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 04:35:38.825910 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 04:35:38.826054 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab.
Entry for principal oryx/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab.
I20260809 04:35:38.848055 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35441
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab
--principal=oryx/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:38.964048 31447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:38.964335 31447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:38.964385 31447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:38.968547 31447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:38.968617 31447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:38.968657 31447 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:35:38.968698 31447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:38.974092 31447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab
--principal=oryx/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.92.62:35441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:38.975169 31447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:38.976222 31447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:38.985888 31458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:38.985881 31454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:38.985880 31453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:38.986141 31447 server_base.cc:1034] running on GCE node
I20260809 04:35:38.986651 31447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:38.987288 31447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:38.988461 31447 hybrid_clock.cc:648] HybridClock initialized: now 1786250138988435 us; error 42 us; skew 500 ppm
Aug 09 04:35:38 dist-test-slave-rmhg krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:38.991364 31447 init.cc:377] Logged in from keytab as oryx/127.19.92.3@KRBTEST.COM (short username oryx)
I20260809 04:35:38.992268 31447 webserver.cc:460] Webserver started at http://127.19.92.3:45237/ using document root <none> and password file <none>
I20260809 04:35:38.992812 31447 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:38.992880 31447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:38.993103 31447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:38.995342 31447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "c41a37869a9f497e8ee518ce57cca8fe"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "a7178885247562d160d62bc6338f7fe9"
server_key_iv: "d2a0992d93e635c30dd877b89b5c88b3"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.995919 31447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "c41a37869a9f497e8ee518ce57cca8fe"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "a7178885247562d160d62bc6338f7fe9"
server_key_iv: "d2a0992d93e635c30dd877b89b5c88b3"
server_key_version: "encryptionkey@0"
I20260809 04:35:38.999761 31447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:35:39.002651 31467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:39.004148 31447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:35:39.004272 31447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "c41a37869a9f497e8ee518ce57cca8fe"
format_stamp: "Formatted at 2026-08-09 04:35:38 on dist-test-slave-rmhg"
server_key: "a7178885247562d160d62bc6338f7fe9"
server_key_iv: "d2a0992d93e635c30dd877b89b5c88b3"
server_key_version: "encryptionkey@0"
I20260809 04:35:39.004388 31447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:39.024056 31447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:39.027470 31447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:39.027655 31447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:39.029137 31447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:39.029222 31447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:39.029278 31447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:39.029302 31447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:39.039937 31447 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:38703
I20260809 04:35:39.040825 31447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:39.041915 31639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:38703 every 8 connection(s)
I20260809 04:35:39.044023 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.030000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:49277 (local address 127.19.92.62:35441)
0809 04:35:39.030158 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.030164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.031022 (+ 858us) server_negotiation.cc:407] Connection header received
0809 04:35:39.032067 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.032071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.032131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.032191 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:39.033579 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.034170 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.035407 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.035728 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.035832 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.038737 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:39.038741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:39.038743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:39.038770 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:39.040283 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.040774 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.040778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.040780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.040860 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.041203 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.041206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.041207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.041429 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:39.041501 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.043698 (+ 2197us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.043840 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:35:39.045076 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.030379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35441 (local address 127.19.92.3:49277)
0809 04:35:39.030859 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.030910 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.031809 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.032533 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.032540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.032866 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:39.033429 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.033446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.034301 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.034304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.035291 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.035302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.035896 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.035901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.036063 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.036637 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:39.036660 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:39.038557 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:39.040401 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:39.040407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:39.040433 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:39.040679 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:39.040945 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:39.040949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:39.040951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:39.041092 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:39.043178 (+ 2086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:39.043184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:39.043454 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.043630 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.043974 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260809 04:35:39.046115 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 31447
I20260809 04:35:39.046253 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:39.046597 31641 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35441
I20260809 04:35:39.046640 19824 external_mini_cluster.cc:1428] Setting key 8d3da2af0e5f48fb4afc01ec19a555c3
I20260809 04:35:39.046754 31641 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:39.047367 31641 heartbeater.cc:511] Master 127.19.92.62:35441 requested a full tablet report, sending...
I20260809 04:35:39.048522 30909 ts_manager.cc:194] Registered new tserver with Master: c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703)
I20260809 04:35:39.049274 30909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:49277
I20260809 04:35:39.055387 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.048704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:50167 (local address 127.19.92.62:35441)
0809 04:35:39.048814 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.048818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.049054 (+ 236us) server_negotiation.cc:407] Connection header received
0809 04:35:39.049126 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.049129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.049197 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.049272 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:39.049664 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.050227 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.051283 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.051496 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.051564 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.052598 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:39.052602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:39.052605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:39.052630 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:39.054101 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.054518 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.054520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.054521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.054591 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.054781 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.054783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.054784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.054938 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:39.055006 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.055186 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.055274 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:35:39.055408 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.048631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35441 (local address 127.19.92.3:50167)
0809 04:35:39.048971 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.048982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.049066 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.049280 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.049282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.049475 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:39.049579 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.049585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.050362 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.050365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.051180 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.051192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.051591 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.051594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.051669 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.052106 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:39.052119 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:39.052457 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:39.054199 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:39.054202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:39.054203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:39.054389 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:39.054650 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:39.054651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:39.054653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:39.054723 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:39.055004 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:39.055006 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:39.055081 (+ 75us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.055149 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.055259 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":98,"threads_started":1}
I20260809 04:35:39.060500 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 04:35:39.086439 30982 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:39.080034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.19.92.62:35441)
0809 04:35:39.080166 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.080170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.080183 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:39.080229 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.080231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.080274 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.080351 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:39.081193 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.081805 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.082748 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.082992 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.083075 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.086288 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56256: BlockingRecv error: recv got EOF from 127.0.0.1:56256 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:39.095789 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.088102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.19.92.62:35441)
0809 04:35:39.088229 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.088234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.088375 (+ 141us) server_negotiation.cc:407] Connection header received
0809 04:35:39.088531 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.088534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.088579 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.088648 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:39.089350 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.089934 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.090821 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.091045 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.091133 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.093383 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:39.093386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:39.093388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:39.093415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:39.094519 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.094876 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.094878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.094879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.094951 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.095134 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.095136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.095137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.095299 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:39.095396 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.095569 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.095663 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:35:39.098585 30909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:39.101696 30909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:39.116792 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.110545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33057 (local address 127.0.0.1:39370)
0809 04:35:39.111625 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.111660 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.111997 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.112552 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.112554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.112570 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:39.112679 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.112686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.114042 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.114046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.115230 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.115240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.116272 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.116276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.116387 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.116429 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.116532 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.116654 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":89,"thread_start_us":103,"threads_started":1}
I20260809 04:35:39.116830 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.111304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.19.92.2:33057)
0809 04:35:39.111626 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.111639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.111808 (+ 169us) server_negotiation.cc:407] Connection header received
0809 04:35:39.112122 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.112127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.112286 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.112407 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:39.112804 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.113956 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.115356 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.116145 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.116248 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.116444 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.116570 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.116678 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":108,"threads_started":1}
I20260809 04:35:39.117233 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.110655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.19.92.1:38877)
0809 04:35:39.111356 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.111364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.111760 (+ 396us) server_negotiation.cc:407] Connection header received
0809 04:35:39.111964 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.111970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.112177 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.112335 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:39.112643 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.113870 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.115583 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.116357 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.116501 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.116636 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.116973 (+ 337us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.117078 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":106,"threads_started":1}
I20260809 04:35:39.117568 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.111171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:38703 (local address 127.0.0.1:56826)
0809 04:35:39.111870 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.111882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.111964 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.112689 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.112692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.112710 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:39.112791 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.112797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.113801 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.113805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.115468 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.115485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.117112 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.117115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.117266 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.117291 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.117366 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.117439 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":51,"thread_start_us":96,"threads_started":1}
I20260809 04:35:39.117622 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.110487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38877 (local address 127.0.0.1:38716)
0809 04:35:39.111067 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.111661 (+ 594us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.111811 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.112330 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.112336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.112362 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:39.112519 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.112526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.114004 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.114008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.115439 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.115450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.116699 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.116703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.116794 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.116816 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.117372 (+ 556us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.117498 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":154,"thread_start_us":82,"threads_started":1}
I20260809 04:35:39.118652 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.111326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.19.92.3:38703)
0809 04:35:39.112211 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.112220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.112240 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:35:39.112312 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.112317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.112540 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.112722 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:39.112885 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.113693 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.115600 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.116150 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.116256 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.116430 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.118322 (+ 1892us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.118497 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":136,"threads_started":1}
I20260809 04:35:39.120100 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 00a6167790db4e488695d9d49144c82d (DEFAULT_TABLE table=test-table [id=a3fa1580fad647d4b0dbff1f717897ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.120318 31115 tablet_service.cc:1467] Processing CreateTablet for tablet 00a6167790db4e488695d9d49144c82d (DEFAULT_TABLE table=test-table [id=a3fa1580fad647d4b0dbff1f717897ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.121311 31115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a6167790db4e488695d9d49144c82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.121312 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a6167790db4e488695d9d49144c82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.121457 31540 tablet_service.cc:1467] Processing CreateTablet for tablet 00a6167790db4e488695d9d49144c82d (DEFAULT_TABLE table=test-table [id=a3fa1580fad647d4b0dbff1f717897ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.122615 31540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a6167790db4e488695d9d49144c82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.127799 31672 tablet_bootstrap.cc:492] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: Bootstrap starting.
I20260809 04:35:39.130095 31672 tablet_bootstrap.cc:654] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.130069 31673 tablet_bootstrap.cc:492] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: Bootstrap starting.
I20260809 04:35:39.131105 31672 log.cc:824] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:39.131155 31674 tablet_bootstrap.cc:492] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: Bootstrap starting.
I20260809 04:35:39.132182 31673 tablet_bootstrap.cc:654] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.133033 31673 log.cc:824] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:39.133080 31674 tablet_bootstrap.cc:654] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.133924 31674 log.cc:824] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:39.133988 31672 tablet_bootstrap.cc:492] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: No bootstrap required, opened a new log
I20260809 04:35:39.134251 31672 ts_tablet_manager.cc:1397] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 04:35:39.135207 31673 tablet_bootstrap.cc:492] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: No bootstrap required, opened a new log
I20260809 04:35:39.135404 31673 ts_tablet_manager.cc:1397] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 04:35:39.135636 31674 tablet_bootstrap.cc:492] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: No bootstrap required, opened a new log
I20260809 04:35:39.135811 31674 ts_tablet_manager.cc:1397] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 04:35:39.138810 31674 raft_consensus.cc:348] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.139052 31674 raft_consensus.cc:374] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.139107 31674 raft_consensus.cc:729] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41a37869a9f497e8ee518ce57cca8fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.139019 31672 raft_consensus.cc:348] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.139293 31672 raft_consensus.cc:374] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.139356 31672 raft_consensus.cc:729] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0311a5fd0fe1452fa045b11c4ec6a6fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.139297 31673 raft_consensus.cc:348] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.139509 31673 raft_consensus.cc:374] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.139554 31673 raft_consensus.cc:729] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dec8009ca4c1dbac87be3221063c0, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.139607 31674 consensus_queue.cc:260] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.139853 31672 consensus_queue.cc:260] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.139890 31673 consensus_queue.cc:260] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.140762 31674 ts_tablet_manager.cc:1428] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:35:39.140796 31673 ts_tablet_manager.cc:1428] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260809 04:35:39.140796 31672 ts_tablet_manager.cc:1428] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:35:39.141167 31220 heartbeater.cc:503] Master 127.19.92.62:35441 was elected leader, sending a full tablet report...
I20260809 04:35:39.141167 31432 heartbeater.cc:503] Master 127.19.92.62:35441 was elected leader, sending a full tablet report...
I20260809 04:35:39.141172 31641 heartbeater.cc:503] Master 127.19.92.62:35441 was elected leader, sending a full tablet report...
I20260809 04:35:39.183238 31684 raft_consensus.cc:482] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:39.183459 31684 raft_consensus.cc:504] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.184669 31684 leader_election.cc:288] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057), c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703)
I20260809 04:35:39.189693 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.185336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:49659 (local address 127.19.92.3:38703)
0809 04:35:39.185460 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.185463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.185478 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:35:39.185567 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.185570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.185624 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.185703 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:39.186142 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.186983 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.188365 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.188960 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.189053 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.189120 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.189478 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.189559 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:39.189693 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.185033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:37749 (local address 127.19.92.2:33057)
0809 04:35:39.185204 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.185209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.185401 (+ 192us) server_negotiation.cc:407] Connection header received
0809 04:35:39.185582 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.185585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.185630 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.185702 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:39.186141 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.186983 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.188331 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.188939 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.189053 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.189120 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.189489 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.189558 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:39.189760 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.184925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33057 (local address 127.19.92.1:37749)
0809 04:35:39.185311 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.185325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.185474 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.185730 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.185732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.185769 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:39.185989 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.185996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.187144 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.187147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.188219 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.188229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.189096 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.189100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.189333 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.189375 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.189455 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.189570 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":103,"spinlock_wait_cycles":34176,"thread_start_us":111,"threads_started":1}
I20260809 04:35:39.189764 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.184994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:38703 (local address 127.19.92.1:49659)
0809 04:35:39.185360 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.185370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.185473 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.185730 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.185732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.185770 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:39.186007 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.186013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.187096 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.187099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.188221 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.188231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.189096 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.189100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.189327 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.189379 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.189458 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.189572 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":207,"spinlock_wait_cycles":8192,"thread_start_us":145,"threads_started":1}
I20260809 04:35:39.190542 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a6167790db4e488695d9d49144c82d" candidate_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dec8009ca4c1dbac87be3221063c0" is_pre_election: true
I20260809 04:35:39.190656 31572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a6167790db4e488695d9d49144c82d" candidate_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41a37869a9f497e8ee518ce57cca8fe" is_pre_election: true
I20260809 04:35:39.190905 31361 raft_consensus.cc:2454] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0311a5fd0fe1452fa045b11c4ec6a6fe in term 0.
I20260809 04:35:39.190905 31572 raft_consensus.cc:2454] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0311a5fd0fe1452fa045b11c4ec6a6fe in term 0.
I20260809 04:35:39.191429 31043 leader_election.cc:302] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311a5fd0fe1452fa045b11c4ec6a6fe, c41a37869a9f497e8ee518ce57cca8fe; no voters:
I20260809 04:35:39.191735 31684 raft_consensus.cc:2781] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:39.191815 31684 raft_consensus.cc:482] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:39.191867 31684 raft_consensus.cc:3037] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.192955 31684 raft_consensus.cc:504] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.193327 31684 leader_election.cc:288] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [CANDIDATE]: Term 1 election: Requested vote from peers 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057), c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703)
I20260809 04:35:39.193641 31572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a6167790db4e488695d9d49144c82d" candidate_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41a37869a9f497e8ee518ce57cca8fe"
I20260809 04:35:39.193681 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a6167790db4e488695d9d49144c82d" candidate_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dec8009ca4c1dbac87be3221063c0"
I20260809 04:35:39.193755 31572 raft_consensus.cc:3037] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.193783 31361 raft_consensus.cc:3037] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.194834 31361 raft_consensus.cc:2454] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0311a5fd0fe1452fa045b11c4ec6a6fe in term 1.
I20260809 04:35:39.195057 31572 raft_consensus.cc:2454] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0311a5fd0fe1452fa045b11c4ec6a6fe in term 1.
I20260809 04:35:39.195134 31047 leader_election.cc:302] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311a5fd0fe1452fa045b11c4ec6a6fe, 4f5dec8009ca4c1dbac87be3221063c0; no voters:
I20260809 04:35:39.195279 31684 raft_consensus.cc:2781] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:39.195497 31684 raft_consensus.cc:686] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 LEADER]: Becoming Leader. State: Replica: 0311a5fd0fe1452fa045b11c4ec6a6fe, State: Running, Role: LEADER
I20260809 04:35:39.195842 31684 consensus_queue.cc:237] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.198985 30906 catalog_manager.cc:5495] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1). New cstate: current_term: 1 leader_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20260809 04:35:39.215337 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.212102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.19.92.1:38877)
0809 04:35:39.212218 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.212222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.212236 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:39.212312 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.212315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.212370 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.212460 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:39.212667 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.213235 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.214048 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.214263 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.214337 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.214565 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:39.215028 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:39.215114 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.215170 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.215223 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:35:39.221382 31572 raft_consensus.cc:1261] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Refusing update from remote peer 0311a5fd0fe1452fa045b11c4ec6a6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:39.221758 31361 raft_consensus.cc:1261] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Refusing update from remote peer 0311a5fd0fe1452fa045b11c4ec6a6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:39.222141 31689 consensus_queue.cc:1035] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:39.222345 31684 consensus_queue.cc:1035] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:39.230870 31697 mvcc.cc:205] Tried to move back new op lower bound from 7316480570243399680 to 7316480570148798464. Current Snapshot: MvccSnapshot[applied={T|T < 7316480570243399680}]
I20260809 04:35:39.239494 30906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:39.239681 30906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:39.242213 30906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: Sending DeleteTablet for 3 replicas of tablet 00a6167790db4e488695d9d49144c82d
I20260809 04:35:39.243093 31304 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a6167790db4e488695d9d49144c82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:39 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:39370
I20260809 04:35:39.243072 31115 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a6167790db4e488695d9d49144c82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:39 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:38716
I20260809 04:35:39.243453 31540 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a6167790db4e488695d9d49144c82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:39 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:56826
I20260809 04:35:39.244021 31713 tablet_replica.cc:330] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: stopping tablet replica
I20260809 04:35:39.244376 31710 tablet_replica.cc:330] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: stopping tablet replica
I20260809 04:35:39.244401 31713 raft_consensus.cc:2229] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:39.244848 31711 tablet_replica.cc:330] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: stopping tablet replica
I20260809 04:35:39.244851 31713 raft_consensus.cc:2258] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:39.244968 30906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 04:35:39.245471 31711 raft_consensus.cc:2229] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 04:35:39.245546 30906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:39.245762 31710 raft_consensus.cc:2229] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:39.246505 31711 raft_consensus.cc:2258] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:39.246718 31713 ts_tablet_manager.cc:1905] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:39.246212 31710 raft_consensus.cc:2258] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:39.248611 31711 ts_tablet_manager.cc:1905] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:39.249558 31710 ts_tablet_manager.cc:1905] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:39.249682 31713 ts_tablet_manager.cc:1918] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:39.249794 31713 log.cc:1196] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/00a6167790db4e488695d9d49144c82d
I20260809 04:35:39.250146 31713 ts_tablet_manager.cc:1939] T 00a6167790db4e488695d9d49144c82d P c41a37869a9f497e8ee518ce57cca8fe: Deleting consensus metadata
I20260809 04:35:39.251308 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 10b6406cae9546a499c7926af5346b9a (DEFAULT_TABLE table=test-table [id=2667979e015a4f6386aeb1ba7b106cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.251569 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b6406cae9546a499c7926af5346b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.253391 30884 catalog_manager.cc:4841] TS c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703): tablet 00a6167790db4e488695d9d49144c82d (table test-table [id=a3fa1580fad647d4b0dbff1f717897ad]) successfully deleted
I20260809 04:35:39.253643 31711 ts_tablet_manager.cc:1918] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:39.253959 31540 tablet_service.cc:1467] Processing CreateTablet for tablet 10b6406cae9546a499c7926af5346b9a (DEFAULT_TABLE table=test-table [id=2667979e015a4f6386aeb1ba7b106cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.254220 31540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b6406cae9546a499c7926af5346b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.255147 31710 ts_tablet_manager.cc:1918] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:39.255231 31710 log.cc:1196] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/00a6167790db4e488695d9d49144c82d
I20260809 04:35:39.255522 31710 ts_tablet_manager.cc:1939] T 00a6167790db4e488695d9d49144c82d P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting consensus metadata
I20260809 04:35:39.255977 31674 tablet_bootstrap.cc:492] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: Bootstrap starting.
I20260809 04:35:39.256588 31115 tablet_service.cc:1467] Processing CreateTablet for tablet 10b6406cae9546a499c7926af5346b9a (DEFAULT_TABLE table=test-table [id=2667979e015a4f6386aeb1ba7b106cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.256614 30887 catalog_manager.cc:4841] TS 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877): tablet 00a6167790db4e488695d9d49144c82d (table test-table [id=a3fa1580fad647d4b0dbff1f717897ad]) successfully deleted
I20260809 04:35:39.253726 31711 log.cc:1196] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/00a6167790db4e488695d9d49144c82d
I20260809 04:35:39.256836 31115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b6406cae9546a499c7926af5346b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.256953 31674 tablet_bootstrap.cc:654] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.257072 31711 ts_tablet_manager.cc:1939] T 00a6167790db4e488695d9d49144c82d P 4f5dec8009ca4c1dbac87be3221063c0: Deleting consensus metadata
I20260809 04:35:39.253904 31673 tablet_bootstrap.cc:492] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: Bootstrap starting.
I20260809 04:35:39.258224 30886 catalog_manager.cc:4841] TS 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057): tablet 00a6167790db4e488695d9d49144c82d (table test-table [id=a3fa1580fad647d4b0dbff1f717897ad]) successfully deleted
I20260809 04:35:39.258522 31673 tablet_bootstrap.cc:654] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.259094 31672 tablet_bootstrap.cc:492] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: Bootstrap starting.
I20260809 04:35:39.260061 31672 tablet_bootstrap.cc:654] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.260460 31674 tablet_bootstrap.cc:492] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: No bootstrap required, opened a new log
I20260809 04:35:39.260560 31674 ts_tablet_manager.cc:1397] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260809 04:35:39.260982 31674 raft_consensus.cc:348] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.261080 31674 raft_consensus.cc:374] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.261101 31674 raft_consensus.cc:729] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41a37869a9f497e8ee518ce57cca8fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.261248 31674 consensus_queue.cc:260] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.261512 31674 ts_tablet_manager.cc:1428] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 04:35:39.262566 31673 tablet_bootstrap.cc:492] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: No bootstrap required, opened a new log
I20260809 04:35:39.262658 31673 ts_tablet_manager.cc:1397] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260809 04:35:39.262814 31672 tablet_bootstrap.cc:492] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: No bootstrap required, opened a new log
I20260809 04:35:39.262907 31672 ts_tablet_manager.cc:1397] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 04:35:39.263094 31673 raft_consensus.cc:348] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.263208 31673 raft_consensus.cc:374] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.263242 31673 raft_consensus.cc:729] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dec8009ca4c1dbac87be3221063c0, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.263360 31673 consensus_queue.cc:260] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.263397 31672 raft_consensus.cc:348] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.263505 31672 raft_consensus.cc:374] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.263536 31672 raft_consensus.cc:729] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0311a5fd0fe1452fa045b11c4ec6a6fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.263600 31673 ts_tablet_manager.cc:1428] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260809 04:35:39.263649 31672 consensus_queue.cc:260] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.263886 31672 ts_tablet_manager.cc:1428] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260809 04:35:39.292146 31642 tablet.cc:2369] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:35:39.312521 31435 tablet.cc:2369] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:35:39.337363 31221 tablet.cc:2369] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:35:39.495877 31703 raft_consensus.cc:482] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:39.496069 31703 raft_consensus.cc:504] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.496560 31703 leader_election.cc:288] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057), c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703)
I20260809 04:35:39.497035 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b6406cae9546a499c7926af5346b9a" candidate_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dec8009ca4c1dbac87be3221063c0" is_pre_election: true
I20260809 04:35:39.497041 31572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b6406cae9546a499c7926af5346b9a" candidate_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41a37869a9f497e8ee518ce57cca8fe" is_pre_election: true
I20260809 04:35:39.497232 31361 raft_consensus.cc:2454] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0311a5fd0fe1452fa045b11c4ec6a6fe in term 0.
I20260809 04:35:39.497272 31572 raft_consensus.cc:2454] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0311a5fd0fe1452fa045b11c4ec6a6fe in term 0.
I20260809 04:35:39.497562 31047 leader_election.cc:302] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311a5fd0fe1452fa045b11c4ec6a6fe, 4f5dec8009ca4c1dbac87be3221063c0; no voters:
I20260809 04:35:39.497746 31703 raft_consensus.cc:2781] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:39.497810 31703 raft_consensus.cc:482] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:39.497855 31703 raft_consensus.cc:3037] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.498863 31703 raft_consensus.cc:504] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.499210 31703 leader_election.cc:288] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [CANDIDATE]: Term 1 election: Requested vote from peers 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057), c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703)
I20260809 04:35:39.499632 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b6406cae9546a499c7926af5346b9a" candidate_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dec8009ca4c1dbac87be3221063c0"
I20260809 04:35:39.499687 31572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b6406cae9546a499c7926af5346b9a" candidate_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41a37869a9f497e8ee518ce57cca8fe"
I20260809 04:35:39.499760 31361 raft_consensus.cc:3037] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.499783 31572 raft_consensus.cc:3037] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.500869 31572 raft_consensus.cc:2454] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0311a5fd0fe1452fa045b11c4ec6a6fe in term 1.
I20260809 04:35:39.500869 31361 raft_consensus.cc:2454] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0311a5fd0fe1452fa045b11c4ec6a6fe in term 1.
I20260809 04:35:39.501147 31043 leader_election.cc:302] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311a5fd0fe1452fa045b11c4ec6a6fe, c41a37869a9f497e8ee518ce57cca8fe; no voters:
I20260809 04:35:39.501327 31703 raft_consensus.cc:2781] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:39.501423 31703 raft_consensus.cc:686] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 LEADER]: Becoming Leader. State: Replica: 0311a5fd0fe1452fa045b11c4ec6a6fe, State: Running, Role: LEADER
I20260809 04:35:39.501554 31703 consensus_queue.cc:237] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.503242 30906 catalog_manager.cc:5495] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1). New cstate: current_term: 1 leader_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } health_report { overall_health: UNKNOWN } } }
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250138, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.62@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:39.592010 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.583232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.19.92.62:35441)
0809 04:35:39.583330 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.583335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.583410 (+ 75us) server_negotiation.cc:407] Connection header received
0809 04:35:39.583523 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.583525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.583568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.583639 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:39.584304 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.585067 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.585878 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.586218 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.586323 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.588860 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:39.588863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:39.588865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:39.588889 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:39.590295 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.590817 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.590819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.590821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.590897 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.591123 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.591125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.591126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.591303 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:39.591385 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.591705 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.591858 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":35}
I20260809 04:35:39.592016 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.582991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35441 (local address 127.0.0.1:56270)
0809 04:35:39.583330 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.583343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.583450 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.583670 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.583672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.584083 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:39.584184 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.584191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.585201 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.585204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.585763 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.585772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.586412 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.586418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.586517 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.587071 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:39.587084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:39.588706 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:39.590418 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:39.590429 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:39.590437 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:39.590732 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:39.590967 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:39.590970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:39.590972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:39.591056 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:39.591389 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:39.591395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:39.591531 (+ 136us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.591637 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.591798 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":93,"threads_started":1}
I20260809 04:35:39.596273 30906 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:56270:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 04:35:39.597034 30906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:39.602135 31115 tablet_service.cc:1467] Processing CreateTablet for tablet 57164f9d939842faae57f6d5bf2eb5a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0590f722040432ab9afb5e56bcf4dc1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:39.602254 31540 tablet_service.cc:1467] Processing CreateTablet for tablet 57164f9d939842faae57f6d5bf2eb5a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0590f722040432ab9afb5e56bcf4dc1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:39.602221 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 57164f9d939842faae57f6d5bf2eb5a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0590f722040432ab9afb5e56bcf4dc1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:35:39.602592 31115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57164f9d939842faae57f6d5bf2eb5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.602617 31540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57164f9d939842faae57f6d5bf2eb5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.602617 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57164f9d939842faae57f6d5bf2eb5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.605327 31672 tablet_bootstrap.cc:492] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Bootstrap starting.
I20260809 04:35:39.605966 31674 tablet_bootstrap.cc:492] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: Bootstrap starting.
I20260809 04:35:39.606207 31672 tablet_bootstrap.cc:654] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.606786 31674 tablet_bootstrap.cc:654] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.607249 31672 tablet_bootstrap.cc:492] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: No bootstrap required, opened a new log
I20260809 04:35:39.607342 31672 ts_tablet_manager.cc:1397] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:39.607731 31673 tablet_bootstrap.cc:492] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: Bootstrap starting.
I20260809 04:35:39.607774 31672 raft_consensus.cc:348] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.607877 31672 raft_consensus.cc:374] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.607904 31672 raft_consensus.cc:729] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0311a5fd0fe1452fa045b11c4ec6a6fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.608047 31674 tablet_bootstrap.cc:492] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: No bootstrap required, opened a new log
I20260809 04:35:39.608037 31672 consensus_queue.cc:260] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.608129 31674 ts_tablet_manager.cc:1397] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 04:35:39.608309 31672 ts_tablet_manager.cc:1428] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:39.608556 31703 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.608580 31674 raft_consensus.cc:348] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.608691 31674 raft_consensus.cc:374] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.608701 31703 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:39.608723 31674 raft_consensus.cc:729] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41a37869a9f497e8ee518ce57cca8fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.608731 31673 tablet_bootstrap.cc:654] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.608836 31674 consensus_queue.cc:260] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.609112 31674 ts_tablet_manager.cc:1428] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 04:35:39.609380 31683 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.609547 31683 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:39.610258 31673 tablet_bootstrap.cc:492] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: No bootstrap required, opened a new log
I20260809 04:35:39.610355 31673 ts_tablet_manager.cc:1397] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260809 04:35:39.610790 31673 raft_consensus.cc:348] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.610909 31673 raft_consensus.cc:374] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.610975 31673 raft_consensus.cc:729] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dec8009ca4c1dbac87be3221063c0, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.611119 31673 consensus_queue.cc:260] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.611415 31673 ts_tablet_manager.cc:1428] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:39.611388 31686 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.611574 31686 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:39.770819 31683 raft_consensus.cc:482] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:39.770978 31683 raft_consensus.cc:504] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.772099 31683 leader_election.cc:288] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877), 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057)
I20260809 04:35:39.777429 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.772474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:56653 (local address 127.19.92.2:33057)
0809 04:35:39.772753 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.772758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.772870 (+ 112us) server_negotiation.cc:407] Connection header received
0809 04:35:39.772977 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.772980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.773029 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.773103 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:39.773399 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.774230 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.776031 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.776684 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.776799 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.776873 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.777189 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.777274 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260809 04:35:39.777467 31729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.772370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33057 (local address 127.19.92.3:56653)
0809 04:35:39.772795 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.772807 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.772889 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.773121 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.773124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.773187 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:39.773283 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.773289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.774333 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.774336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.775891 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.775905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.776801 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.776805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.777009 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.777056 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.777130 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.777280 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":212,"spinlock_wait_cycles":1920,"thread_start_us":64,"threads_started":1}
I20260809 04:35:39.777467 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.772341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38877 (local address 127.19.92.3:59075)
0809 04:35:39.772684 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.772700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.772813 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.773088 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.773091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.773111 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:39.773283 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.773289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.774333 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.774336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.775891 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.775905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.776801 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.776804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.777019 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.777056 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.777130 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.777280 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":259,"thread_start_us":85,"threads_started":1}
I20260809 04:35:39.777429 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.772473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:59075 (local address 127.19.92.1:38877)
0809 04:35:39.772754 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.772758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.772775 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:35:39.772920 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.772923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.772981 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.773080 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:39.773399 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.774230 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.776031 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.776693 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.776799 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.776873 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.777185 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.777274 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260809 04:35:39.777998 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57164f9d939842faae57f6d5bf2eb5a1" candidate_uuid: "c41a37869a9f497e8ee518ce57cca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dec8009ca4c1dbac87be3221063c0" is_pre_election: true
I20260809 04:35:39.778144 31361 raft_consensus.cc:2454] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c41a37869a9f497e8ee518ce57cca8fe in term 0.
I20260809 04:35:39.778280 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57164f9d939842faae57f6d5bf2eb5a1" candidate_uuid: "c41a37869a9f497e8ee518ce57cca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" is_pre_election: true
I20260809 04:35:39.778549 31141 raft_consensus.cc:2454] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c41a37869a9f497e8ee518ce57cca8fe in term 0.
I20260809 04:35:39.778541 31470 leader_election.cc:302] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5dec8009ca4c1dbac87be3221063c0, c41a37869a9f497e8ee518ce57cca8fe; no voters:
I20260809 04:35:39.778801 31683 raft_consensus.cc:2781] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:39.778859 31683 raft_consensus.cc:482] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:39.778913 31683 raft_consensus.cc:3037] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.779985 31683 raft_consensus.cc:504] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.780331 31683 leader_election.cc:288] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [CANDIDATE]: Term 1 election: Requested vote from peers 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877), 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057)
I20260809 04:35:39.780700 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57164f9d939842faae57f6d5bf2eb5a1" candidate_uuid: "c41a37869a9f497e8ee518ce57cca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe"
I20260809 04:35:39.780704 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57164f9d939842faae57f6d5bf2eb5a1" candidate_uuid: "c41a37869a9f497e8ee518ce57cca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dec8009ca4c1dbac87be3221063c0"
I20260809 04:35:39.780824 31361 raft_consensus.cc:3037] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.780825 31141 raft_consensus.cc:3037] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:39.781993 31141 raft_consensus.cc:2454] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c41a37869a9f497e8ee518ce57cca8fe in term 1.
I20260809 04:35:39.781993 31361 raft_consensus.cc:2454] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c41a37869a9f497e8ee518ce57cca8fe in term 1.
I20260809 04:35:39.782302 31471 leader_election.cc:302] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311a5fd0fe1452fa045b11c4ec6a6fe, c41a37869a9f497e8ee518ce57cca8fe; no voters:
I20260809 04:35:39.782485 31683 raft_consensus.cc:2781] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:39.782658 31683 raft_consensus.cc:686] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [term 1 LEADER]: Becoming Leader. State: Replica: c41a37869a9f497e8ee518ce57cca8fe, State: Running, Role: LEADER
I20260809 04:35:39.782904 31683 consensus_queue.cc:237] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.784152 31734 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: TxnStatusTablet state changed. Reason: New leader c41a37869a9f497e8ee518ce57cca8fe. Latest consensus state: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.784267 31734 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:39.784823 31739 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 04:35:39.784971 31739 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 04:35:39.785471 30909 catalog_manager.cc:5495] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3). New cstate: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } health_report { overall_health: HEALTHY } } }
I20260809 04:35:39.800693 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.796489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.19.92.3:38703)
0809 04:35:39.796771 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.796777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.796791 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:39.796862 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.796864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.796918 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.797013 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:39.797325 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.797896 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.799137 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.799394 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.799496 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.799834 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:39.800304 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:39.800392 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.800493 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.800551 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":121,"threads_started":1}
I20260809 04:35:39.800733 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.796364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:38703 (local address 127.0.0.1:53006)
0809 04:35:39.796563 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.796580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.796689 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.797044 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.797047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.797058 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:39.797199 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.797207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.797999 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.798003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.798995 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.799009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.799509 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.799513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.799609 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.799675 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:39.800392 (+ 717us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:39.800406 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.800471 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.800585 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 04:35:39.804493 31361 raft_consensus.cc:1261] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Refusing update from remote peer c41a37869a9f497e8ee518ce57cca8fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:39.804667 31141 raft_consensus.cc:1261] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Refusing update from remote peer c41a37869a9f497e8ee518ce57cca8fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:39.805032 31683 consensus_queue.cc:1035] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:39.805297 31734 consensus_queue.cc:1035] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:39.807248 31686 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: TxnStatusTablet state changed. Reason: New leader c41a37869a9f497e8ee518ce57cca8fe. Latest consensus state: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.807377 31686 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:39.807812 31703 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: TxnStatusTablet state changed. Reason: New leader c41a37869a9f497e8ee518ce57cca8fe. Latest consensus state: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.807930 31703 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:39.808768 31734 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.808877 31734 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:39.810626 31734 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.810731 31734 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P c41a37869a9f497e8ee518ce57cca8fe: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:35:39.811486 31686 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.811594 31686 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P 4f5dec8009ca4c1dbac87be3221063c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:39.811568 31703 tablet_replica.cc:439] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } } }
I20260809 04:35:39.811676 31703 tablet_replica.cc:442] T 57164f9d939842faae57f6d5bf2eb5a1 P 0311a5fd0fe1452fa045b11c4ec6a6fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:35:39.815968 31750 mvcc.cc:205] Tried to move back new op lower bound from 7316480572632522752 to 7316480572553408512. Current Snapshot: MvccSnapshot[applied={T|T < 7316480572632522752}]
I20260809 04:35:39.816133 31748 mvcc.cc:205] Tried to move back new op lower bound from 7316480572632522752 to 7316480572553408512. Current Snapshot: MvccSnapshot[applied={T|T < 7316480572632522752 or (T in {7316480572632522752})}]
I20260809 04:35:39.830019 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.826643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:45965 (local address 127.19.92.3:38703)
0809 04:35:39.826882 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.826886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.826938 (+ 52us) server_negotiation.cc:407] Connection header received
0809 04:35:39.827094 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.827098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.827147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.827209 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:39.827455 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.827983 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.828891 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.829124 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.829244 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.829577 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:39.829657 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:39.829716 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.829813 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.829871 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260809 04:35:39.830026 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.826545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:38703 (local address 127.19.92.1:45965)
0809 04:35:39.826859 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.826875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.826995 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.827233 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.827235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.827244 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:39.827351 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.827356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.828115 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.828119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.828772 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.828782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.829319 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.829322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.829392 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.829441 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:39.829732 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:39.829739 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.829796 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.829888 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":112,"threads_started":1}
I20260809 04:35:39.837260 31572 raft_consensus.cc:1261] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Refusing update from remote peer 0311a5fd0fe1452fa045b11c4ec6a6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:39.837260 31361 raft_consensus.cc:1261] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Refusing update from remote peer 0311a5fd0fe1452fa045b11c4ec6a6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:39.837695 31703 consensus_queue.cc:1035] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:39.837881 31689 consensus_queue.cc:1035] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:39.862573 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.858958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:48549 (local address 127.19.92.1:38877)
0809 04:35:39.859080 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.859084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.859275 (+ 191us) server_negotiation.cc:407] Connection header received
0809 04:35:39.859408 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.859411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.859467 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.859543 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:39.859842 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.860445 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.861539 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.861769 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.861861 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.862150 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:39.862223 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:39.862281 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.862377 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.862432 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:35:39.862614 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.858837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38877 (local address 127.19.92.3:48549)
0809 04:35:39.859193 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:39.859208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:39.859315 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:39.859567 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:39.859570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:39.859577 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:39.859707 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.859714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.860560 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.860563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.861425 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:39.861438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.861877 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.861880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:39.861976 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.862025 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:39.862304 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:39.862310 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:35:39.862368 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 04:35:39.862468 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":129,"threads_started":1}
I20260809 04:35:39.888408 30909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:39.888584 30909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:39.890841 30909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: Sending DeleteTablet for 3 replicas of tablet 10b6406cae9546a499c7926af5346b9a
I20260809 04:35:39.891522 31304 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b6406cae9546a499c7926af5346b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:39 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:39370
I20260809 04:35:39.891566 31115 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b6406cae9546a499c7926af5346b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:39 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:38716
I20260809 04:35:39.891906 31776 tablet_replica.cc:330] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: stopping tablet replica
I20260809 04:35:39.892050 31540 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b6406cae9546a499c7926af5346b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:39 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:56826
I20260809 04:35:39.892083 31777 tablet_replica.cc:330] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: stopping tablet replica
I20260809 04:35:39.892102 31776 raft_consensus.cc:2229] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:39.892201 31777 raft_consensus.cc:2229] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:39.892364 31776 raft_consensus.cc:2258] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:39.892405 31777 raft_consensus.cc:2258] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:39.893278 31776 ts_tablet_manager.cc:1905] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:39.896241 31776 ts_tablet_manager.cc:1918] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:39.896340 31776 log.cc:1196] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/10b6406cae9546a499c7926af5346b9a
I20260809 04:35:39.896605 31776 ts_tablet_manager.cc:1939] T 10b6406cae9546a499c7926af5346b9a P 4f5dec8009ca4c1dbac87be3221063c0: Deleting consensus metadata
I20260809 04:35:39.897400 30886 catalog_manager.cc:4841] TS 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057): tablet 10b6406cae9546a499c7926af5346b9a (table test-table [id=2667979e015a4f6386aeb1ba7b106cac]) successfully deleted
I20260809 04:35:39.897673 31777 ts_tablet_manager.cc:1905] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:39.899667 31779 tablet_replica.cc:330] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: stopping tablet replica
I20260809 04:35:39.899821 31779 raft_consensus.cc:2229] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:39.899986 31779 raft_consensus.cc:2258] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:39.900731 31779 ts_tablet_manager.cc:1905] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:39.900749 31777 ts_tablet_manager.cc:1918] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:39.900825 31777 log.cc:1196] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/10b6406cae9546a499c7926af5346b9a
I20260809 04:35:39.901134 31777 ts_tablet_manager.cc:1939] T 10b6406cae9546a499c7926af5346b9a P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting consensus metadata
I20260809 04:35:39.901953 30887 catalog_manager.cc:4841] TS 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877): tablet 10b6406cae9546a499c7926af5346b9a (table test-table [id=2667979e015a4f6386aeb1ba7b106cac]) successfully deleted
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260809 04:35:39.908005 31779 ts_tablet_manager.cc:1918] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:39.908107 31779 log.cc:1196] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/10b6406cae9546a499c7926af5346b9a
I20260809 04:35:39.908427 31779 ts_tablet_manager.cc:1939] T 10b6406cae9546a499c7926af5346b9a P c41a37869a9f497e8ee518ce57cca8fe: Deleting consensus metadata
I20260809 04:35:39.909248 30884 catalog_manager.cc:4841] TS c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703): tablet 10b6406cae9546a499c7926af5346b9a (table test-table [id=2667979e015a4f6386aeb1ba7b106cac]) successfully deleted
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 04:35:39.924048 30982 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:39.917205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.19.92.62:35441)
0809 04:35:39.917341 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.917346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.917623 (+ 277us) server_negotiation.cc:407] Connection header received
0809 04:35:39.917741 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.917744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.917795 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.917877 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:39.918628 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.919173 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.920140 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.920388 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.920486 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.923897 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45586: BlockingRecv error: recv got EOF from 127.0.0.1:45586 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 09 04:35:39 dist-test-slave-rmhg krb5kdc[30843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM
I20260809 04:35:39.935014 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:39.925766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.19.92.62:35441)
0809 04:35:39.925902 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:39.925906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:39.925927 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:35:39.926057 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:39.926061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:39.926118 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:39.926227 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:39.926777 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.927603 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.928425 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:39.928787 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:39.928901 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:39.931444 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:39.931448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:39.931450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:39.931484 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:39.933103 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.933606 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.933610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.933612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.933716 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:39.934042 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:39.934046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:39.934048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:39.934303 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:39.934449 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:39.934710 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 04:35:39.934844 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:35:39.937742 30909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:35:39.938205 30909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:39.942759 31115 tablet_service.cc:1467] Processing CreateTablet for tablet b711417437f14dd29c7823c207a7bc27 (DEFAULT_TABLE table=test-table [id=2894503c8e344bdf9694129e7cb61a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.942895 31304 tablet_service.cc:1467] Processing CreateTablet for tablet b711417437f14dd29c7823c207a7bc27 (DEFAULT_TABLE table=test-table [id=2894503c8e344bdf9694129e7cb61a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.942911 31540 tablet_service.cc:1467] Processing CreateTablet for tablet b711417437f14dd29c7823c207a7bc27 (DEFAULT_TABLE table=test-table [id=2894503c8e344bdf9694129e7cb61a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:39.943074 31115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b711417437f14dd29c7823c207a7bc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.943184 31540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b711417437f14dd29c7823c207a7bc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.943183 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b711417437f14dd29c7823c207a7bc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:39.945544 31674 tablet_bootstrap.cc:492] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: Bootstrap starting.
I20260809 04:35:39.945542 31673 tablet_bootstrap.cc:492] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: Bootstrap starting.
I20260809 04:35:39.945565 31672 tablet_bootstrap.cc:492] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Bootstrap starting.
I20260809 04:35:39.946421 31672 tablet_bootstrap.cc:654] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.946484 31674 tablet_bootstrap.cc:654] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.946487 31673 tablet_bootstrap.cc:654] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:39.947983 31672 tablet_bootstrap.cc:492] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: No bootstrap required, opened a new log
I20260809 04:35:39.948081 31672 ts_tablet_manager.cc:1397] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 04:35:39.948464 31674 tablet_bootstrap.cc:492] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: No bootstrap required, opened a new log
I20260809 04:35:39.948462 31672 raft_consensus.cc:348] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.948545 31674 ts_tablet_manager.cc:1397] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 04:35:39.948567 31672 raft_consensus.cc:374] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.948598 31672 raft_consensus.cc:729] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0311a5fd0fe1452fa045b11c4ec6a6fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.948712 31672 consensus_queue.cc:260] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.948824 31673 tablet_bootstrap.cc:492] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: No bootstrap required, opened a new log
I20260809 04:35:39.948897 31673 ts_tablet_manager.cc:1397] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:35:39.948946 31672 ts_tablet_manager.cc:1428] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:39.948995 31674 raft_consensus.cc:348] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.949105 31674 raft_consensus.cc:374] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.949134 31674 raft_consensus.cc:729] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41a37869a9f497e8ee518ce57cca8fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.949306 31674 consensus_queue.cc:260] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.949558 31674 ts_tablet_manager.cc:1428] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:39.949625 31673 raft_consensus.cc:348] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.949718 31673 raft_consensus.cc:374] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:39.949747 31673 raft_consensus.cc:729] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dec8009ca4c1dbac87be3221063c0, State: Initialized, Role: FOLLOWER
I20260809 04:35:39.949862 31673 consensus_queue.cc:260] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:39.950095 31673 ts_tablet_manager.cc:1428] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:40.020416 31686 raft_consensus.cc:482] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:40.020577 31686 raft_consensus.cc:504] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.021836 31686 leader_election.cc:288] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877), c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703)
I20260809 04:35:40.025888 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:40.022260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:57815 (local address 127.19.92.1:38877)
0809 04:35:40.022461 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:40.022465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:40.022600 (+ 135us) server_negotiation.cc:407] Connection header received
0809 04:35:40.022744 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:40.022747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:40.022790 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:40.022857 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:40.023139 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.023627 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.024817 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.025296 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.025393 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:40.025467 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:40.025685 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 04:35:40.025757 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 04:35:40.025897 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:40.022219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:51883 (local address 127.19.92.3:38703)
0809 04:35:40.022346 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:40.022351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:40.022548 (+ 197us) server_negotiation.cc:407] Connection header received
0809 04:35:40.022692 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:40.022695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:40.022737 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:40.022814 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:35:40.023109 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.023650 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.024847 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.025306 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.025406 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:40.025477 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:40.025690 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 04:35:40.025770 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:35:40.025941 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:40.022087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:38703 (local address 127.19.92.2:51883)
0809 04:35:40.022452 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:40.022465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:40.022572 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:40.022840 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:40.022842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:40.022859 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:40.023003 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:40.023009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.023753 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.023757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:40.024720 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:40.024730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.025418 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.025422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:40.025581 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:40.025599 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 04:35:40.025664 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 04:35:40.025748 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":74,"spinlock_wait_cycles":13568,"thread_start_us":137,"threads_started":1}
I20260809 04:35:40.026103 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:40.022087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38877 (local address 127.19.92.2:57815)
0809 04:35:40.022521 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:40.022535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:40.022645 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:40.022866 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:40.022869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:40.022910 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:35:40.023029 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:40.023034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.023745 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.023748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:40.024708 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:40.024717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.025406 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.025409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:40.025569 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:40.025597 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 04:35:40.025663 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 04:35:40.025744 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":143,"spinlock_wait_cycles":2816,"thread_start_us":97,"threads_started":1}
I20260809 04:35:40.026460 31572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b711417437f14dd29c7823c207a7bc27" candidate_uuid: "4f5dec8009ca4c1dbac87be3221063c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41a37869a9f497e8ee518ce57cca8fe" is_pre_election: true
I20260809 04:35:40.026623 31572 raft_consensus.cc:2454] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5dec8009ca4c1dbac87be3221063c0 in term 0.
I20260809 04:35:40.026665 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b711417437f14dd29c7823c207a7bc27" candidate_uuid: "4f5dec8009ca4c1dbac87be3221063c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" is_pre_election: true
I20260809 04:35:40.026827 31141 raft_consensus.cc:2454] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5dec8009ca4c1dbac87be3221063c0 in term 0.
I20260809 04:35:40.026981 31257 leader_election.cc:302] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5dec8009ca4c1dbac87be3221063c0, c41a37869a9f497e8ee518ce57cca8fe; no voters:
I20260809 04:35:40.027251 31686 raft_consensus.cc:2781] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:40.027313 31686 raft_consensus.cc:482] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:40.027356 31686 raft_consensus.cc:3037] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:40.028405 31686 raft_consensus.cc:504] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.028746 31686 leader_election.cc:288] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [CANDIDATE]: Term 1 election: Requested vote from peers 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877), c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703)
I20260809 04:35:40.029065 31572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b711417437f14dd29c7823c207a7bc27" candidate_uuid: "4f5dec8009ca4c1dbac87be3221063c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41a37869a9f497e8ee518ce57cca8fe"
I20260809 04:35:40.029062 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b711417437f14dd29c7823c207a7bc27" candidate_uuid: "4f5dec8009ca4c1dbac87be3221063c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe"
I20260809 04:35:40.029197 31141 raft_consensus.cc:3037] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:40.029199 31572 raft_consensus.cc:3037] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:40.030158 31141 raft_consensus.cc:2454] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5dec8009ca4c1dbac87be3221063c0 in term 1.
I20260809 04:35:40.030157 31572 raft_consensus.cc:2454] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5dec8009ca4c1dbac87be3221063c0 in term 1.
I20260809 04:35:40.030414 31257 leader_election.cc:302] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5dec8009ca4c1dbac87be3221063c0, c41a37869a9f497e8ee518ce57cca8fe; no voters:
I20260809 04:35:40.030589 31686 raft_consensus.cc:2781] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:40.030797 31686 raft_consensus.cc:686] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4f5dec8009ca4c1dbac87be3221063c0, State: Running, Role: LEADER
I20260809 04:35:40.031052 31686 consensus_queue.cc:237] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.033237 30906 catalog_manager.cc:5495] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2). New cstate: current_term: 1 leader_uuid: "4f5dec8009ca4c1dbac87be3221063c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20260809 04:35:40.056654 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:40.052585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.19.92.2:33057)
0809 04:35:40.052722 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:40.052727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:40.052743 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:35:40.052857 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:40.052859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:40.052928 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:40.053037 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:40.053289 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.054107 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.054987 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.055336 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.055429 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:40.055637 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:40.056233 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:40.056341 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:40.056450 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 04:35:40.056515 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:35:40.059433 31141 raft_consensus.cc:1261] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Refusing update from remote peer 4f5dec8009ca4c1dbac87be3221063c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:40.059499 31572 raft_consensus.cc:1261] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Refusing update from remote peer 4f5dec8009ca4c1dbac87be3221063c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:40.059911 31686 consensus_queue.cc:1035] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:40.060115 31811 consensus_queue.cc:1035] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:40.071393 30906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:40.071537 30906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:40.073438 30906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: Sending DeleteTablet for 3 replicas of tablet b711417437f14dd29c7823c207a7bc27
I20260809 04:35:40.073946 31115 tablet_service.cc:1514] Processing DeleteTablet for tablet b711417437f14dd29c7823c207a7bc27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:40 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:38716
I20260809 04:35:40.074018 31304 tablet_service.cc:1514] Processing DeleteTablet for tablet b711417437f14dd29c7823c207a7bc27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:40 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:39370
I20260809 04:35:40.074131 31777 tablet_replica.cc:330] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: stopping tablet replica
I20260809 04:35:40.074159 31540 tablet_service.cc:1514] Processing DeleteTablet for tablet b711417437f14dd29c7823c207a7bc27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:40 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:56826
I20260809 04:35:40.074242 31777 raft_consensus.cc:2229] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:40.074323 31776 tablet_replica.cc:330] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: stopping tablet replica
I20260809 04:35:40.074368 31777 raft_consensus.cc:2258] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:40.074420 31776 raft_consensus.cc:2229] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:40.074538 31779 tablet_replica.cc:330] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: stopping tablet replica
I20260809 04:35:40.074646 31779 raft_consensus.cc:2229] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:40.074685 31776 raft_consensus.cc:2258] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:40.074764 31779 raft_consensus.cc:2258] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:40.075418 31776 ts_tablet_manager.cc:1905] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:40.075476 31777 ts_tablet_manager.cc:1905] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:40.077445 30906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 04:35:40.077906 31776 ts_tablet_manager.cc:1918] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:40.077906 31777 ts_tablet_manager.cc:1918] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:40.078001 31777 log.cc:1196] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/b711417437f14dd29c7823c207a7bc27
I20260809 04:35:40.078001 31776 log.cc:1196] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/b711417437f14dd29c7823c207a7bc27
I20260809 04:35:40.078253 31777 ts_tablet_manager.cc:1939] T b711417437f14dd29c7823c207a7bc27 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Deleting consensus metadata
I20260809 04:35:40.078253 31776 ts_tablet_manager.cc:1939] T b711417437f14dd29c7823c207a7bc27 P 4f5dec8009ca4c1dbac87be3221063c0: Deleting consensus metadata
W20260809 04:35:40.078259 30906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:35:40.078886 30886 catalog_manager.cc:4841] TS 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057): tablet b711417437f14dd29c7823c207a7bc27 (table test-table [id=2894503c8e344bdf9694129e7cb61a96]) successfully deleted
I20260809 04:35:40.078886 30887 catalog_manager.cc:4841] TS 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877): tablet b711417437f14dd29c7823c207a7bc27 (table test-table [id=2894503c8e344bdf9694129e7cb61a96]) successfully deleted
I20260809 04:35:40.079716 31779 ts_tablet_manager.cc:1905] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:40.082268 31779 ts_tablet_manager.cc:1918] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:35:40.082362 31779 log.cc:1196] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/b711417437f14dd29c7823c207a7bc27
I20260809 04:35:40.082629 31779 ts_tablet_manager.cc:1939] T b711417437f14dd29c7823c207a7bc27 P c41a37869a9f497e8ee518ce57cca8fe: Deleting consensus metadata
I20260809 04:35:40.083036 31115 tablet_service.cc:1467] Processing CreateTablet for tablet 18513433db8b4862b154d7bb4bba6cf2 (DEFAULT_TABLE table=test-table [id=fe0bb467a5c54d7daaa1a68bcbacb0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:40.083241 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 18513433db8b4862b154d7bb4bba6cf2 (DEFAULT_TABLE table=test-table [id=fe0bb467a5c54d7daaa1a68bcbacb0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:40.083315 31115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18513433db8b4862b154d7bb4bba6cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:40.083474 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18513433db8b4862b154d7bb4bba6cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:40.083890 30884 catalog_manager.cc:4841] TS c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703): tablet b711417437f14dd29c7823c207a7bc27 (table test-table [id=2894503c8e344bdf9694129e7cb61a96]) successfully deleted
I20260809 04:35:40.084949 31540 tablet_service.cc:1467] Processing CreateTablet for tablet 18513433db8b4862b154d7bb4bba6cf2 (DEFAULT_TABLE table=test-table [id=fe0bb467a5c54d7daaa1a68bcbacb0a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:35:40.085210 31540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18513433db8b4862b154d7bb4bba6cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:40.085726 31673 tablet_bootstrap.cc:492] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: Bootstrap starting.
I20260809 04:35:40.085726 31672 tablet_bootstrap.cc:492] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Bootstrap starting.
I20260809 04:35:40.086534 31672 tablet_bootstrap.cc:654] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:40.086596 31673 tablet_bootstrap.cc:654] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:40.087386 31674 tablet_bootstrap.cc:492] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: Bootstrap starting.
I20260809 04:35:40.087874 31673 tablet_bootstrap.cc:492] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: No bootstrap required, opened a new log
I20260809 04:35:40.087953 31673 ts_tablet_manager.cc:1397] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:40.088241 31674 tablet_bootstrap.cc:654] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:40.088402 31673 raft_consensus.cc:348] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.088532 31673 raft_consensus.cc:374] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:40.088570 31673 raft_consensus.cc:729] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dec8009ca4c1dbac87be3221063c0, State: Initialized, Role: FOLLOWER
I20260809 04:35:40.088697 31673 consensus_queue.cc:260] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.088974 31673 ts_tablet_manager.cc:1428] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:40.089407 31674 tablet_bootstrap.cc:492] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: No bootstrap required, opened a new log
I20260809 04:35:40.089502 31674 ts_tablet_manager.cc:1397] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:40.089898 31674 raft_consensus.cc:348] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.090029 31674 raft_consensus.cc:374] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:40.090060 31674 raft_consensus.cc:729] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41a37869a9f497e8ee518ce57cca8fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:40.090080 31672 tablet_bootstrap.cc:492] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe: No bootstrap required, opened a new log
I20260809 04:35:40.090154 31672 ts_tablet_manager.cc:1397] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 04:35:40.090179 31674 consensus_queue.cc:260] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.090453 31674 ts_tablet_manager.cc:1428] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:40.090559 31672 raft_consensus.cc:348] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.090682 31672 raft_consensus.cc:374] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:40.090718 31672 raft_consensus.cc:729] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0311a5fd0fe1452fa045b11c4ec6a6fe, State: Initialized, Role: FOLLOWER
I20260809 04:35:40.090826 31672 consensus_queue.cc:260] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.091064 31672 ts_tablet_manager.cc:1428] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 04:35:40.137404 31683 raft_consensus.cc:482] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:35:40.137573 31683 raft_consensus.cc:504] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.138023 31683 leader_election.cc:288] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877), 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057)
I20260809 04:35:40.138446 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18513433db8b4862b154d7bb4bba6cf2" candidate_uuid: "c41a37869a9f497e8ee518ce57cca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" is_pre_election: true
I20260809 04:35:40.138479 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18513433db8b4862b154d7bb4bba6cf2" candidate_uuid: "c41a37869a9f497e8ee518ce57cca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dec8009ca4c1dbac87be3221063c0" is_pre_election: true
I20260809 04:35:40.138597 31141 raft_consensus.cc:2454] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c41a37869a9f497e8ee518ce57cca8fe in term 0.
I20260809 04:35:40.138631 31361 raft_consensus.cc:2454] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c41a37869a9f497e8ee518ce57cca8fe in term 0.
I20260809 04:35:40.138904 31471 leader_election.cc:302] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311a5fd0fe1452fa045b11c4ec6a6fe, c41a37869a9f497e8ee518ce57cca8fe; no voters:
I20260809 04:35:40.139114 31683 raft_consensus.cc:2781] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:35:40.139199 31683 raft_consensus.cc:482] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:35:40.139247 31683 raft_consensus.cc:3037] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:40.140210 31683 raft_consensus.cc:504] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.140527 31683 leader_election.cc:288] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [CANDIDATE]: Term 1 election: Requested vote from peers 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877), 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057)
I20260809 04:35:40.140869 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18513433db8b4862b154d7bb4bba6cf2" candidate_uuid: "c41a37869a9f497e8ee518ce57cca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe"
I20260809 04:35:40.140890 31361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18513433db8b4862b154d7bb4bba6cf2" candidate_uuid: "c41a37869a9f497e8ee518ce57cca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dec8009ca4c1dbac87be3221063c0"
I20260809 04:35:40.140985 31141 raft_consensus.cc:3037] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:40.141004 31361 raft_consensus.cc:3037] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:35:40.141927 31141 raft_consensus.cc:2454] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c41a37869a9f497e8ee518ce57cca8fe in term 1.
I20260809 04:35:40.142149 31361 raft_consensus.cc:2454] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c41a37869a9f497e8ee518ce57cca8fe in term 1.
I20260809 04:35:40.142216 31471 leader_election.cc:302] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0311a5fd0fe1452fa045b11c4ec6a6fe, c41a37869a9f497e8ee518ce57cca8fe; no voters:
I20260809 04:35:40.142565 31683 raft_consensus.cc:2781] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:40.142665 31683 raft_consensus.cc:686] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 1 LEADER]: Becoming Leader. State: Replica: c41a37869a9f497e8ee518ce57cca8fe, State: Running, Role: LEADER
I20260809 04:35:40.142807 31683 consensus_queue.cc:237] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } }
I20260809 04:35:40.144471 30906 catalog_manager.cc:5495] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3). New cstate: current_term: 1 leader_uuid: "c41a37869a9f497e8ee518ce57cca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c41a37869a9f497e8ee518ce57cca8fe" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 38703 } health_report { overall_health: HEALTHY } } }
I20260809 04:35:40.161302 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:40.158064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.19.92.3:38703)
0809 04:35:40.158196 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:40.158200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:40.158215 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:35:40.158260 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:40.158263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:40.158315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:40.158391 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:40.158622 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.159370 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.160214 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.160533 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.160639 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:40.160839 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:40.160933 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:40.161004 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:40.161061 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 04:35:40.161121 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:35:40.170434 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:40.167338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:38703 (local address 127.19.92.3:50685)
0809 04:35:40.167448 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:40.167460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:40.167527 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:40.167743 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:40.167745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:40.167751 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:35:40.167833 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:40.167837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.168756 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.168758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:40.169367 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:40.169374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.169900 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.169902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:40.169956 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:40.169980 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:35:40.170209 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:35:40.170213 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:35:40.170249 (+ 36us) client_negotiation.cc:240] Negotiation successful
0809 04:35:40.170313 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260809 04:35:40.170455 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:40.167408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:50685 (local address 127.19.92.3:38703)
0809 04:35:40.167552 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:40.167556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:40.167570 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:40.167612 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:40.167614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:40.167662 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:40.167741 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:35:40.167916 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.168662 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.169482 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:40.169810 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:40.169911 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:40.170070 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:35:40.170147 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:35:40.170211 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:40.170273 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 04:35:40.170320 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:35:40.175882 31141 raft_consensus.cc:1261] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Refusing update from remote peer c41a37869a9f497e8ee518ce57cca8fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:40.175881 31361 raft_consensus.cc:1261] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Refusing update from remote peer c41a37869a9f497e8ee518ce57cca8fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:35:40.176302 31762 consensus_queue.cc:1035] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0311a5fd0fe1452fa045b11c4ec6a6fe" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:40.176476 31683 consensus_queue.cc:1035] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5dec8009ca4c1dbac87be3221063c0" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:35:40.219255 30906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:40.219449 30906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:35:40.222071 30906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8442cfa7de411ea293f57331d8091a: Sending DeleteTablet for 3 replicas of tablet 18513433db8b4862b154d7bb4bba6cf2
I20260809 04:35:40.222762 31539 tablet_service.cc:1514] Processing DeleteTablet for tablet 18513433db8b4862b154d7bb4bba6cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:40 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:56826
I20260809 04:35:40.222945 31779 tablet_replica.cc:330] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: stopping tablet replica
I20260809 04:35:40.223058 31779 raft_consensus.cc:2229] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:35:40.223073 31115 tablet_service.cc:1514] Processing DeleteTablet for tablet 18513433db8b4862b154d7bb4bba6cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:40 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:38716
I20260809 04:35:40.223290 31304 tablet_service.cc:1514] Processing DeleteTablet for tablet 18513433db8b4862b154d7bb4bba6cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:35:40 UTC) from {username='oryx', principal='oryx/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:39370
I20260809 04:35:40.223376 31779 raft_consensus.cc:2258] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:40.223466 31777 tablet_replica.cc:330] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe: stopping tablet replica
I20260809 04:35:40.223582 31777 raft_consensus.cc:2229] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:40.223702 31777 raft_consensus.cc:2258] T 18513433db8b4862b154d7bb4bba6cf2 P 0311a5fd0fe1452fa045b11c4ec6a6fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:40.223872 31776 tablet_replica.cc:330] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: stopping tablet replica
I20260809 04:35:40.224004 31776 raft_consensus.cc:2229] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:35:40.224116 31776 raft_consensus.cc:2258] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:35:40.224888 31779 ts_tablet_manager.cc:1905] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:40.226749 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 31018
I20260809 04:35:40.229956 31779 ts_tablet_manager.cc:1918] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:40.230103 31779 log.cc:1196] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/18513433db8b4862b154d7bb4bba6cf2
I20260809 04:35:40.230373 31779 ts_tablet_manager.cc:1939] T 18513433db8b4862b154d7bb4bba6cf2 P c41a37869a9f497e8ee518ce57cca8fe: Deleting consensus metadata
I20260809 04:35:40.234427 30884 catalog_manager.cc:4841] TS c41a37869a9f497e8ee518ce57cca8fe (127.19.92.3:38703): tablet 18513433db8b4862b154d7bb4bba6cf2 (table test-table [id=fe0bb467a5c54d7daaa1a68bcbacb0a8]) successfully deleted
I20260809 04:35:40.238035 31776 ts_tablet_manager.cc:1905] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:35:40.240658 31776 ts_tablet_manager.cc:1918] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:35:40.240737 31776 log.cc:1196] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/18513433db8b4862b154d7bb4bba6cf2
I20260809 04:35:40.240986 31776 ts_tablet_manager.cc:1939] T 18513433db8b4862b154d7bb4bba6cf2 P 4f5dec8009ca4c1dbac87be3221063c0: Deleting consensus metadata
I20260809 04:35:40.241925 30886 catalog_manager.cc:4841] TS 4f5dec8009ca4c1dbac87be3221063c0 (127.19.92.2:33057): tablet 18513433db8b4862b154d7bb4bba6cf2 (table test-table [id=fe0bb467a5c54d7daaa1a68bcbacb0a8]) successfully deleted
W20260809 04:35:40.244042 30887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:38877 (error 108)
W20260809 04:35:40.245262 31830 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:40.244792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38877 (local address 127.0.0.1:38166)
0809 04:35:40.245040 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:40.245119 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:38877: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":59,"threads_started":1}
I20260809 04:35:40.245407 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 31233
W20260809 04:35:40.252426 30887 catalog_manager.cc:4584] TS 0311a5fd0fe1452fa045b11c4ec6a6fe (127.19.92.1:38877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18513433db8b4862b154d7bb4bba6cf2: Network error: Client connection negotiation failed: client connection to 127.19.92.1:38877: connect: Connection reset by peer (error 104)
I20260809 04:35:40.261018 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 31447
I20260809 04:35:40.275403 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 30861
2026-08-09T04:35:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3449 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:40 dist-test-slave-rmhg krb5kdc[31833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:40 dist-test-slave-rmhg krb5kdc[31833](info): set up 2 sockets
Aug 09 04:35:40 dist-test-slave-rmhg krb5kdc[31833](info): commencing operation
krb5kdc: starting...
W20260809 04:35:41.411655 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:41 dist-test-slave-rmhg krb5kdc[31833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:41.536581 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:44953
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:44953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:41.648510 31849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:41.648811 31849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:41.648869 31849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:41.653019 31849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:41.653087 31849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:41.653115 31849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:41.653134 31849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:41.653177 31849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:41.658574 31849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:44953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:41.659644 31849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:41.660714 31849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:35:41.669350 31849 server_base.cc:1034] running on GCE node
W20260809 04:35:41.669305 31856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:41.669334 31855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:41.669314 31860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:41.670058 31849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:41.671180 31849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:41.672341 31849 hybrid_clock.cc:648] HybridClock initialized: now 1786250141672327 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T04:35:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1413 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:35:41 dist-test-slave-rmhg krb5kdc[31870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:35:41 dist-test-slave-rmhg krb5kdc[31870](info): set up 2 sockets
Aug 09 04:35:41 dist-test-slave-rmhg krb5kdc[31870](info): commencing operation
krb5kdc: starting...
W20260809 04:35:42.829030 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:35:42 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:35:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:35:42.957798 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:38445
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:38445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:43.072044 31888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:43.072368 31888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:43.072431 31888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:43.076517 31888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:43.076593 31888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:43.076630 31888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:43.076653 31888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:43.076678 31888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:43.082206 31888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:38445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:38445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:43.083428 31888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:43.084582 31888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:43.093024 31898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:43.093215 31888 server_base.cc:1034] running on GCE node
W20260809 04:35:43.093043 31895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:43.093014 31896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:43.094035 31888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:43.095124 31888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:43.096288 31888 hybrid_clock.cc:648] HybridClock initialized: now 1786250143096273 us; error 27 us; skew 500 ppm
Aug 09 04:35:43 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:43.098982 31888 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:43.099884 31888 webserver.cc:460] Webserver started at http://127.19.92.62:39471/ using document root <none> and password file <none>
I20260809 04:35:43.100431 31888 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:43.100498 31888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:43.100692 31888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:43.102768 31888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "1785e912227247a48d08c9b76595e2c9"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "b063f7c8108d4e2763a1a016cbfc851c"
server_key_iv: "372c6fbca133e259a1888d96cd0b3eec"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.103330 31888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "1785e912227247a48d08c9b76595e2c9"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "b063f7c8108d4e2763a1a016cbfc851c"
server_key_iv: "372c6fbca133e259a1888d96cd0b3eec"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.107007 31888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 04:35:43.109360 31911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.110548 31888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:43.110664 31888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "1785e912227247a48d08c9b76595e2c9"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "b063f7c8108d4e2763a1a016cbfc851c"
server_key_iv: "372c6fbca133e259a1888d96cd0b3eec"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.110761 31888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:43.154759 31888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:43.157935 31888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:43.158130 31888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:43.164937 31888 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:38445
I20260809 04:35:43.164953 31981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:38445 every 8 connection(s)
I20260809 04:35:43.165915 31888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:43.168840 31984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:35:43.174419 31984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Bootstrap starting.
I20260809 04:35:43.175817 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 31888
I20260809 04:35:43.175971 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:35:43.176273 19824 external_mini_cluster.cc:1428] Setting key 9a49dde23aa7640d498b8a3ce1d6af36
I20260809 04:35:43.177253 31984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Neither blocks nor log segments found. Creating new log.
I20260809 04:35:43.178172 31984 log.cc:824] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:43.180052 31984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: No bootstrap required, opened a new log
I20260809 04:35:43.183590 31984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } }
I20260809 04:35:43.183845 31984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:35:43.183907 31984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1785e912227247a48d08c9b76595e2c9, State: Initialized, Role: FOLLOWER
I20260809 04:35:43.184376 31984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } }
I20260809 04:35:43.184574 31984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:43.184648 31984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:43.184755 31984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 04:35:43 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:43.185726 31984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } }
I20260809 04:35:43.186108 31984 leader_election.cc:302] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1785e912227247a48d08c9b76595e2c9; no voters:
I20260809 04:35:43.186430 31984 leader_election.cc:288] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:35:43.186547 31997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:35:43.186789 31997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 1 LEADER]: Becoming Leader. State: Replica: 1785e912227247a48d08c9b76595e2c9, State: Running, Role: LEADER
I20260809 04:35:43.187160 31997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } }
I20260809 04:35:43.187819 31984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:43.189029 31998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1785e912227247a48d08c9b76595e2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } } }
I20260809 04:35:43.189169 31998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:43.189574 32006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:43.189525 31999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1785e912227247a48d08c9b76595e2c9. Latest consensus state: current_term: 1 leader_uuid: "1785e912227247a48d08c9b76595e2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } } }
I20260809 04:35:43.190999 31999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:43.192452 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:43.177514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.19.92.62:38445)
0809 04:35:43.177974 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:43.177989 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:43.178024 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:35:43.178720 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:43.178739 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:43.179067 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:43.179384 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:43.180228 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.181260 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.182105 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.182584 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.182740 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:43.185479 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:43.185484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:43.185498 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:43.185527 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:43.187485 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:43.188061 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:43.188068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:43.188074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:43.188242 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:43.189694 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:43.189697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:43.189699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:43.190232 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:43.191025 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:43.191205 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 04:35:43.191509 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260809 04:35:43.192904 32006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:43.198457 32006 catalog_manager.cc:1347] Generated new cluster ID: dd9f1edb24884becac2f41b90f6bbd59
I20260809 04:35:43.198527 32006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:43.212592 32006 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:35:43.213665 32006 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:43.224992 32006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Generated new TSK 0
I20260809 04:35:43.225793 32006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:35:43.261942 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38445
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:43.378922 32023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:43.379192 32023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:43.379236 32023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:43.383172 32023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:43.383229 32023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:43.383284 32023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:43.388559 32023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:43.389668 32023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:43.390683 32023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:43.400125 32031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:43.400166 32030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:43.400125 32039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:43.400424 32023 server_base.cc:1034] running on GCE node
I20260809 04:35:43.400884 32023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:43.401719 32023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:43.402899 32023 hybrid_clock.cc:648] HybridClock initialized: now 1786250143402879 us; error 34 us; skew 500 ppm
Aug 09 04:35:43 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:43.405745 32023 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:35:43.406690 32023 webserver.cc:460] Webserver started at http://127.19.92.1:45421/ using document root <none> and password file <none>
I20260809 04:35:43.407284 32023 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:43.407351 32023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:43.407552 32023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:43.409775 32023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "be8906d836124dbca8ff594ff201a709"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "14c0bc7ae87d05d79b20c5bb16ec9b5f"
server_key_iv: "fbc64590e622db3bad9cbca7d7babf86"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.410331 32023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "be8906d836124dbca8ff594ff201a709"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "14c0bc7ae87d05d79b20c5bb16ec9b5f"
server_key_iv: "fbc64590e622db3bad9cbca7d7babf86"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.414142 32023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:35:43.416613 32048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.417834 32023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:43.417949 32023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "be8906d836124dbca8ff594ff201a709"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "14c0bc7ae87d05d79b20c5bb16ec9b5f"
server_key_iv: "fbc64590e622db3bad9cbca7d7babf86"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.418043 32023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:43.455794 32023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:43.458917 32023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:43.459146 32023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:43.459822 32023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:43.460703 32023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:43.460753 32023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.460808 32023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:43.460839 32023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.469216 32023 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:40553
I20260809 04:35:43.469244 32225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:40553 every 8 connection(s)
I20260809 04:35:43.470165 32023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
Aug 09 04:35:43 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:43.480098 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 32023
I20260809 04:35:43.480259 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:35:43.480600 19824 external_mini_cluster.cc:1428] Setting key 3eea9650c2572ffdb10aef913cc6b175
I20260809 04:35:43.483748 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:43.471995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:44529 (local address 127.19.92.62:38445)
0809 04:35:43.472125 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:43.472128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:43.473190 (+ 1062us) server_negotiation.cc:407] Connection header received
0809 04:35:43.474049 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:43.474051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:43.474100 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:43.474176 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:43.475363 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.475904 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.477094 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.477358 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.477430 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:43.479996 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:43.480000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:43.480003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:43.480035 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:43.481513 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:43.482089 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:43.482092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:43.482094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:43.482179 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:43.482518 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:43.482521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:43.482522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:43.482781 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:43.482878 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:43.483421 (+ 543us) server_negotiation.cc:299] Negotiation successful
0809 04:35:43.483568 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:43.484644 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:43.472320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:44529)
0809 04:35:43.473037 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:43.473080 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:43.473866 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:43.474333 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:43.474340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:43.474699 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:43.475213 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:43.475227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.476041 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.476044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:43.476988 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:43.476999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.477514 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.477517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:43.477677 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:43.478276 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:43.478293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:43.479786 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:43.481623 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:43.481629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:43.481649 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:43.482001 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:43.482268 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:43.482271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:43.482273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:43.482429 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:43.482886 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:43.482892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:43.483182 (+ 290us) client_negotiation.cc:769] Sending connection context
0809 04:35:43.483352 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 04:35:43.483649 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":122,"threads_started":1}
I20260809 04:35:43.485898 32226 heartbeater.cc:347] Connected to a master server at 127.19.92.62:38445
I20260809 04:35:43.486162 32226 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:43.486677 32226 heartbeater.cc:511] Master 127.19.92.62:38445 requested a full tablet report, sending...
I20260809 04:35:43.488562 31936 ts_manager.cc:194] Registered new tserver with Master: be8906d836124dbca8ff594ff201a709 (127.19.92.1:40553)
I20260809 04:35:43.490209 31936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:44529
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:35:43.509449 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38445
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:43.620764 32233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:43.621049 32233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:43.621094 32233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:43.625247 32233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:43.625310 32233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:43.625381 32233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:43.630739 32233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:43.631856 32233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:43.632915 32233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:43.642539 32246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:43.642515 32240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:43.642632 32243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:43.642892 32233 server_base.cc:1034] running on GCE node
I20260809 04:35:43.643337 32233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:43.644032 32233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:43.645213 32233 hybrid_clock.cc:648] HybridClock initialized: now 1786250143645193 us; error 35 us; skew 500 ppm
Aug 09 04:35:43 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:43.648062 32233 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:35:43.648986 32233 webserver.cc:460] Webserver started at http://127.19.92.2:33561/ using document root <none> and password file <none>
I20260809 04:35:43.649583 32233 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:43.649653 32233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:43.649866 32233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:43.651957 32233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "62f091bedb1e4033ad98e4f2550a491a"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "33b211ee0a8355d67b215c2abb4db44f"
server_key_iv: "4962f5b8b3d29c2bcdd84e931c539e15"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.652515 32233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "62f091bedb1e4033ad98e4f2550a491a"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "33b211ee0a8355d67b215c2abb4db44f"
server_key_iv: "4962f5b8b3d29c2bcdd84e931c539e15"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.656220 32233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:35:43.658695 32256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.659883 32233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:35:43.660000 32233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "62f091bedb1e4033ad98e4f2550a491a"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "33b211ee0a8355d67b215c2abb4db44f"
server_key_iv: "4962f5b8b3d29c2bcdd84e931c539e15"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.660106 32233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:43.684020 32233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:43.687373 32233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:43.687544 32233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:43.688170 32233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:43.689057 32233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:43.689116 32233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.689194 32233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:43.689220 32233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.697902 32233 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:43647
I20260809 04:35:43.697922 32421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:43647 every 8 connection(s)
I20260809 04:35:43.698773 32233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:43.706557 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 32233
I20260809 04:35:43.706692 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:35:43.707015 19824 external_mini_cluster.cc:1428] Setting key 19983bc420a97ffc510b760091679e65
Aug 09 04:35:43 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:43.712077 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:43.700609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:43805 (local address 127.19.92.62:38445)
0809 04:35:43.700738 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:43.700742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:43.701768 (+ 1026us) server_negotiation.cc:407] Connection header received
0809 04:35:43.702658 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:43.702661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:43.702721 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:43.702794 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:43.704001 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.704549 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.705744 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.705964 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.706064 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:43.708845 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:43.708848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:43.708850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:43.708874 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:43.710144 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:43.710620 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:43.710623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:43.710624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:43.710703 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:43.711021 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:43.711024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:43.711026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:43.711204 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:43.711288 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:43.711786 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 04:35:43.711908 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:43.712862 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:43.700916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:43805)
0809 04:35:43.701614 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:43.701657 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:43.702447 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:43.702986 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:43.702993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:43.703346 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:43.703852 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:43.703866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.704668 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.704671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:43.705619 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:43.705629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.706105 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.706110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:43.706292 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:43.706907 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:43.706929 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:43.708692 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:43.710252 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:43.710258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:43.710274 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:43.710527 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:43.710788 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:43.710791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:43.710792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:43.710929 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:43.711303 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:43.711310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:43.711561 (+ 251us) client_negotiation.cc:769] Sending connection context
0809 04:35:43.711725 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 04:35:43.712044 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":113,"threads_started":1}
I20260809 04:35:43.713963 32423 heartbeater.cc:347] Connected to a master server at 127.19.92.62:38445
I20260809 04:35:43.714233 32423 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:43.714891 32423 heartbeater.cc:511] Master 127.19.92.62:38445 requested a full tablet report, sending...
I20260809 04:35:43.716130 31936 ts_manager.cc:194] Registered new tserver with Master: 62f091bedb1e4033ad98e4f2550a491a (127.19.92.2:43647)
I20260809 04:35:43.716902 31936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:43805
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:35:43.736002 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38445
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:43.851334 32430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:43.851606 32430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:43.851650 32430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:43.855655 32430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:43.855711 32430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:43.855769 32430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:43.860920 32430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:43.861958 32430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:43.862985 32430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:43.872402 32437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:43.872408 32436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:43.872404 32439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:43.872550 32430 server_base.cc:1034] running on GCE node
I20260809 04:35:43.873014 32430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:43.873701 32430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:43.874879 32430 hybrid_clock.cc:648] HybridClock initialized: now 1786250143874853 us; error 35 us; skew 500 ppm
Aug 09 04:35:43 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:43.877535 32430 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:35:43.878358 32430 webserver.cc:460] Webserver started at http://127.19.92.3:44675/ using document root <none> and password file <none>
I20260809 04:35:43.878880 32430 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:43.878937 32430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:43.879109 32430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:35:43.881126 32430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "8c0cafac6f8747ddae48a042d543f3a3"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "1276087872da428cfa83520e7d3018b2"
server_key_iv: "60bcfb3f3c118d333ac5a2dc245e29c2"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.881735 32430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "8c0cafac6f8747ddae48a042d543f3a3"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "1276087872da428cfa83520e7d3018b2"
server_key_iv: "60bcfb3f3c118d333ac5a2dc245e29c2"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.885793 32430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:35:43.888366 32446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.889698 32430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:35:43.889835 32430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "8c0cafac6f8747ddae48a042d543f3a3"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "1276087872da428cfa83520e7d3018b2"
server_key_iv: "60bcfb3f3c118d333ac5a2dc245e29c2"
server_key_version: "encryptionkey@0"
I20260809 04:35:43.889941 32430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:43.906360 32430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:43.909179 32430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:43.909369 32430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:43.909983 32430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:43.910837 32430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:43.910898 32430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.910969 32430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:43.911001 32430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:43.920262 32430 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:42685
I20260809 04:35:43.920313 32626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:42685 every 8 connection(s)
I20260809 04:35:43.921188 32430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:43.923257 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 32430
I20260809 04:35:43.923408 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:35:43.923730 19824 external_mini_cluster.cc:1428] Setting key 385c225258f068a6d0a97824571a3298
Aug 09 04:35:43 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:35:43.934659 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:43.923095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:55447 (local address 127.19.92.62:38445)
0809 04:35:43.924195 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:43.924201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:43.924688 (+ 487us) server_negotiation.cc:407] Connection header received
0809 04:35:43.925819 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:43.925822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:43.925867 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:43.925923 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:43.927173 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.927695 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.928773 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.928978 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.929052 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:43.931527 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:43.931531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:43.931533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:43.931557 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:43.932707 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:43.933214 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:43.933218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:43.933219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:43.933295 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:43.933604 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:43.933608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:43.933610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:43.933788 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:43.933874 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:43.934373 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 04:35:43.934494 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983}
I20260809 04:35:43.935439 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:43.923650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:55447)
0809 04:35:43.924523 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:43.924584 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:43.925603 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:43.926149 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:43.926158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:43.926540 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:43.927053 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:43.927067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.927788 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.927791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:43.928682 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:43.928690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:43.929133 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:43.929136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:43.929345 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:43.929927 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:43.929943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:43.931368 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:35:43.932815 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:43.932820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:43.932836 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:43.933083 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:43.933380 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:35:43.933383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:35:43.933385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:35:43.933520 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:35:43.933887 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:35:43.933894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:35:43.934147 (+ 253us) client_negotiation.cc:769] Sending connection context
0809 04:35:43.934314 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 04:35:43.934649 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":151,"threads_started":1}
I20260809 04:35:43.936522 32627 heartbeater.cc:347] Connected to a master server at 127.19.92.62:38445
I20260809 04:35:43.936764 32627 heartbeater.cc:464] Registering TS with master...
I20260809 04:35:43.937326 32627 heartbeater.cc:511] Master 127.19.92.62:38445 requested a full tablet report, sending...
I20260809 04:35:43.938310 31936 ts_manager.cc:194] Registered new tserver with Master: 8c0cafac6f8747ddae48a042d543f3a3 (127.19.92.3:42685)
I20260809 04:35:43.938820 31936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:55447
I20260809 04:35:43.949724 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab.
Entry for principal oryx/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab.
WARNING: no policy specified for oryx/127.19.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab.
Entry for principal oryx/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab.
WARNING: no policy specified for oryx/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab.
Entry for principal oryx/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab.
I20260809 04:35:44.025023 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 32023
I20260809 04:35:44.037912 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 32233
I20260809 04:35:44.051614 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 32430
I20260809 04:35:44.058524 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 31888
I20260809 04:35:44.077981 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:38445
--webserver_interface=127.19.92.62
--webserver_port=39471
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:38445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:35:44.190909 32636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:44.191196 32636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:44.191236 32636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:44.195255 32636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:35:44.195317 32636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:44.195335 32636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:44.195350 32636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:35:44.195364 32636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:35:44.200616 32636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:38445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:38445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=39471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:44.201725 32636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:44.202790 32636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:35:44.211737 32636 server_base.cc:1034] running on GCE node
W20260809 04:35:44.211692 32649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:44.211692 32643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:44.211699 32642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:44.212435 32636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:44.213639 32636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:44.214792 32636 hybrid_clock.cc:648] HybridClock initialized: now 1786250144214765 us; error 45 us; skew 500 ppm
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250144, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:44.217821 32636 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:35:44.218747 32636 webserver.cc:460] Webserver started at http://127.19.92.62:39471/ using document root <none> and password file <none>
I20260809 04:35:44.219290 32636 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:44.219363 32636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:44.223672 32636 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:35:44.225975 32660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.227236 32636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:35:44.227360 32636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "1785e912227247a48d08c9b76595e2c9"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "b063f7c8108d4e2763a1a016cbfc851c"
server_key_iv: "372c6fbca133e259a1888d96cd0b3eec"
server_key_version: "encryptionkey@0"
I20260809 04:35:44.227916 32636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:44.242712 32636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:44.246093 32636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:44.246268 32636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:44.252950 32636 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:38445
I20260809 04:35:44.253010 32741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:38445 every 8 connection(s)
I20260809 04:35:44.254047 32636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:35:44.254848 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 32636
I20260809 04:35:44.255275 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:40553
--local_ip_for_outbound_sockets=127.19.92.1
--tserver_master_addrs=127.19.92.62:38445
--webserver_port=45421
--webserver_interface=127.19.92.1
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.92.1
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:35:44.258555 32742 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:35:44.260314 32742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Bootstrap starting.
I20260809 04:35:44.276983 32742 log.cc:824] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:35:44.283842 32742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:35:44.284241 32742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Bootstrap complete.
I20260809 04:35:44.288448 32742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } }
I20260809 04:35:44.288890 32742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1785e912227247a48d08c9b76595e2c9, State: Initialized, Role: FOLLOWER
I20260809 04:35:44.289431 32742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } }
I20260809 04:35:44.289564 32742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:35:44.289635 32742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:35:44.289737 32742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:35:44.291745 32742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } }
I20260809 04:35:44.292083 32742 leader_election.cc:302] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1785e912227247a48d08c9b76595e2c9; no voters:
I20260809 04:35:44.292403 32742 leader_election.cc:288] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 04:35:44.292521 32747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:35:44.292719 32747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [term 2 LEADER]: Becoming Leader. State: Replica: 1785e912227247a48d08c9b76595e2c9, State: Running, Role: LEADER
I20260809 04:35:44.293098 32747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } }
I20260809 04:35:44.293764 32742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:35:44.294656 32752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1785e912227247a48d08c9b76595e2c9. Latest consensus state: current_term: 2 leader_uuid: "1785e912227247a48d08c9b76595e2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } } }
I20260809 04:35:44.294699 32751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1785e912227247a48d08c9b76595e2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1785e912227247a48d08c9b76595e2c9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 38445 } } }
I20260809 04:35:44.294780 32752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:44.294811 32751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:35:44.298995 304 catalog_manager.cc:1259] Loaded cluster ID: dd9f1edb24884becac2f41b90f6bbd59
I20260809 04:35:44.299059 304 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: loading cluster ID for follower catalog manager: success
I20260809 04:35:44.301523 304 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: acquiring CA information for follower catalog manager: success
I20260809 04:35:44.302951 304 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:35:44.303076 32764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:35:44.303915 32764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:35:44.304358 32764 catalog_manager.cc:1259] Loaded cluster ID: dd9f1edb24884becac2f41b90f6bbd59
I20260809 04:35:44.304438 32764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:35:44.305132 32764 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:35:44.305653 32764 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1785e912227247a48d08c9b76595e2c9: Loaded TSK: 0
I20260809 04:35:44.306437 32764 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:35:44.370416 32745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:44.370707 32745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:44.370776 32745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:44.374934 32745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:44.375005 32745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:44.375088 32745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:35:44.382468 32745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.1.keytab
--principal=oryx/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:40553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=45421
--tserver_master_addrs=127.19.92.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:44.383538 32745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:44.384572 32745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:44.394140 312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:44.394140 313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:44.394228 32745 server_base.cc:1034] running on GCE node
W20260809 04:35:44.394397 315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:44.394933 32745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:44.395781 32745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:44.397002 32745 hybrid_clock.cc:648] HybridClock initialized: now 1786250144396975 us; error 37 us; skew 500 ppm
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250144, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:44.400436 32745 init.cc:377] Logged in from keytab as oryx/127.19.92.1@KRBTEST.COM (short username oryx)
I20260809 04:35:44.401721 32745 webserver.cc:460] Webserver started at http://127.19.92.1:45421/ using document root <none> and password file <none>
I20260809 04:35:44.402522 32745 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:44.402642 32745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:44.408380 32745 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:35:44.410689 326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.412070 32745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:35:44.412181 32745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "be8906d836124dbca8ff594ff201a709"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "14c0bc7ae87d05d79b20c5bb16ec9b5f"
server_key_iv: "fbc64590e622db3bad9cbca7d7babf86"
server_key_version: "encryptionkey@0"
I20260809 04:35:44.412719 32745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:44.429359 32745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:44.432670 32745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:44.432855 32745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:44.433487 32745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:44.434372 32745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:44.434420 32745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.434475 32745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:44.434505 32745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.443303 32745 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:40553
I20260809 04:35:44.443327 505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:40553 every 8 connection(s)
I20260809 04:35:44.444496 32745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:35:44.452816 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 32745
I20260809 04:35:44.453322 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:43647
--local_ip_for_outbound_sockets=127.19.92.2
--tserver_master_addrs=127.19.92.62:38445
--webserver_port=33561
--webserver_interface=127.19.92.2
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.92.2
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:44.458196 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.446901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:42893)
0809 04:35:44.447458 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.447507 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.448779 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.449994 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.450003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.450448 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.450940 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.450955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.452267 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.452270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.453266 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.453277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.454121 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.454125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.454314 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.454907 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.454930 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.457390 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":136,"threads_started":1}
W20260809 04:35:44.458657 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.446815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:42893 (local address 127.19.92.62:38445)
0809 04:35:44.447371 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.447392 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.447631 (+ 239us) server_negotiation.cc:407] Connection header received
0809 04:35:44.449235 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.449246 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.449558 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.449874 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.451120 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.452122 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.453390 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.453952 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.454123 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.457679 (+ 3556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:42893: BlockingRecv error: recv got EOF from 127.19.92.1:42893 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":150,"threads_started":1}
W20260809 04:35:44.458799 506 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:38445 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.92.62:38445: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:44.464092 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.459233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:34443)
0809 04:35:44.459345 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.459360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.459438 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.459800 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.459803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.459948 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.460044 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.460049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.460758 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.460761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.461357 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.461366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.461783 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.461786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.461875 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.462355 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.462366 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.463971 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 04:35:44.464231 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.459488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:34443 (local address 127.19.92.62:38445)
0809 04:35:44.459613 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.459616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.459626 (+ 10us) server_negotiation.cc:407] Connection header received
0809 04:35:44.459658 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.459660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.459698 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.459765 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.460151 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.460659 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.461471 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.461687 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.461771 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.464127 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:34443: BlockingRecv error: recv got EOF from 127.19.92.1:34443 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:44.469628 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.464780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:51609)
0809 04:35:44.464952 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.464965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.465032 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.465256 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.465258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.465371 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.465443 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.465447 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.466394 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.466396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.466893 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.466900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.467399 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.467401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.467453 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.467880 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.467890 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.469538 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260809 04:35:44.469796 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.464894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:51609 (local address 127.19.92.62:38445)
0809 04:35:44.465013 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.465017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.465031 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:44.465094 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.465097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.465139 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.465244 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.465533 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.466312 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.466981 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.467315 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.467406 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.469686 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:51609: BlockingRecv error: recv got EOF from 127.19.92.1:51609 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 04:35:44.469821 506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:35:44.569028 512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:44.569355 512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:44.569404 512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:44.573463 512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:44.573537 512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:44.573597 512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:35:44.579030 512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.2.keytab
--principal=oryx/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:43647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=33561
--tserver_master_addrs=127.19.92.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:44.580101 512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:44.581094 512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:44.590796 524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:44.591005 522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:44.590801 520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:44.591079 512 server_base.cc:1034] running on GCE node
I20260809 04:35:44.591553 512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:44.592191 512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:44.593334 512 hybrid_clock.cc:648] HybridClock initialized: now 1786250144593303 us; error 47 us; skew 500 ppm
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250144, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:44.596143 512 init.cc:377] Logged in from keytab as oryx/127.19.92.2@KRBTEST.COM (short username oryx)
I20260809 04:35:44.597002 512 webserver.cc:460] Webserver started at http://127.19.92.2:33561/ using document root <none> and password file <none>
I20260809 04:35:44.597571 512 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:44.597659 512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:44.601603 512 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:35:44.603777 533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.605201 512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:44.605329 512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "62f091bedb1e4033ad98e4f2550a491a"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "33b211ee0a8355d67b215c2abb4db44f"
server_key_iv: "4962f5b8b3d29c2bcdd84e931c539e15"
server_key_version: "encryptionkey@0"
I20260809 04:35:44.605892 512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:44.635725 512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:44.638659 512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:44.638844 512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:44.639452 512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:44.640313 512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:44.640362 512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.640417 512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:44.640447 512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.649211 512 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:43647
I20260809 04:35:44.649259 717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:43647 every 8 connection(s)
I20260809 04:35:44.650272 512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:35:44.651340 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 512
I20260809 04:35:44.651738 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:42685
--local_ip_for_outbound_sockets=127.19.92.3
--tserver_master_addrs=127.19.92.62:38445
--webserver_port=44675
--webserver_interface=127.19.92.3
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.92.3
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 04:35:44.661793 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.652192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:49091 (local address 127.19.92.62:38445)
0809 04:35:44.652352 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.652357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.653347 (+ 990us) server_negotiation.cc:407] Connection header received
0809 04:35:44.654757 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.654761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.654823 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.654898 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.656125 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.656668 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.657841 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.658075 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.658166 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.661684 (+ 3518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:49091: BlockingRecv error: recv got EOF from 127.19.92.2:49091 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:35:44.662595 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.652664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:49091)
0809 04:35:44.653188 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.653243 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.654535 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.655092 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.655099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.655476 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.655962 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.655986 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.656791 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.656794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.657714 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.657723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.658216 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.658220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.658387 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.658957 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.658982 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.661537 (+ 2555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
W20260809 04:35:44.663089 718 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:38445 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.92.62:38445: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:44.668216 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.663504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:41277)
0809 04:35:44.663658 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.663671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.663744 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.663959 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.663962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.664117 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.664207 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.664212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.664950 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.664953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.665535 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.665544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.665950 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.665954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.666051 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.666471 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.666483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.668060 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260809 04:35:44.668625 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.663614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:41277 (local address 127.19.92.62:38445)
0809 04:35:44.663733 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.663737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.663751 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:44.663818 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.663829 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.663870 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.663933 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.664314 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.664849 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.665644 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.665855 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.665940 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.668322 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:41277: BlockingRecv error: recv got EOF from 127.19.92.2:41277 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:44.673131 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.668716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:47113)
0809 04:35:44.668867 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.668880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.668963 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.669188 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.669191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.669323 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.669390 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.669395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.670075 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.670078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.670578 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.670585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.670980 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.670984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.671058 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.671448 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.671460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.673023 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260809 04:35:44.673261 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.668816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:47113 (local address 127.19.92.62:38445)
0809 04:35:44.668915 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.668918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.668932 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:44.669042 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.669044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.669083 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.669143 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.669500 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.669974 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.670677 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.670886 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.670968 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.673134 (+ 2166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:47113: BlockingRecv error: recv got EOF from 127.19.92.2:47113 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 04:35:44.673466 718 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:35:44.768736 724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:35:44.769018 724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:35:44.769064 724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:35:44.773137 724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:35:44.773239 724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:35:44.773304 724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:35:44.778964 724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/krb5kdc/oryx_127.19.92.3.keytab
--principal=oryx/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:42685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=44675
--tserver_master_addrs=127.19.92.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:35:44.780095 724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:35:44.781095 724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:35:44.790827 732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:44.790894 737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:35:44.790827 734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:35:44.791138 724 server_base.cc:1034] running on GCE node
I20260809 04:35:44.791536 724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:35:44.792171 724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:35:44.793325 724 hybrid_clock.cc:648] HybridClock initialized: now 1786250144793306 us; error 31 us; skew 500 ppm
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250144, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:35:44.796171 724 init.cc:377] Logged in from keytab as oryx/127.19.92.3@KRBTEST.COM (short username oryx)
I20260809 04:35:44.797065 724 webserver.cc:460] Webserver started at http://127.19.92.3:44675/ using document root <none> and password file <none>
I20260809 04:35:44.797657 724 fs_manager.cc:359] Metadata directory not provided
I20260809 04:35:44.797740 724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:35:44.801753 724 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:35:44.803934 746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.805347 724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:35:44.805475 724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "8c0cafac6f8747ddae48a042d543f3a3"
format_stamp: "Formatted at 2026-08-09 04:35:43 on dist-test-slave-rmhg"
server_key: "1276087872da428cfa83520e7d3018b2"
server_key_iv: "60bcfb3f3c118d333ac5a2dc245e29c2"
server_key_version: "encryptionkey@0"
I20260809 04:35:44.806066 724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:35:44.816164 724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:35:44.819130 724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:35:44.819311 724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:35:44.819934 724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:35:44.820837 724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:35:44.820888 724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.820943 724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:35:44.820968 724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:35:44.830741 724 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:42685
I20260809 04:35:44.830773 919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:42685 every 8 connection(s)
I20260809 04:35:44.831851 724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:35:44.839229 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 724
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 04:35:44.844328 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.833821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:48151 (local address 127.19.92.62:38445)
0809 04:35:44.833951 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.833955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.834903 (+ 948us) server_negotiation.cc:407] Connection header received
0809 04:35:44.836324 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.836327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.836392 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.836469 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.837885 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.838431 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.840003 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.840311 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.840427 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.844191 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:48151: BlockingRecv error: recv got EOF from 127.19.92.3:48151 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:35:44.845278 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.834231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:48151)
0809 04:35:44.834738 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.834785 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.836079 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.836693 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.836702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.837116 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.837721 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.837738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.838568 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.838571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.839841 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.839857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.840496 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.840502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.840813 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.841496 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.841517 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.844024 (+ 2507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
W20260809 04:35:44.845878 920 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:38445 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.92.62:38445: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
I20260809 04:35:44.850081 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.840387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.19.92.62:38445)
0809 04:35:44.840637 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.840640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.840650 (+ 10us) server_negotiation.cc:407] Connection header received
0809 04:35:44.840677 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.840679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.840713 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.840786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.841817 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.842355 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.843342 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.843579 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.843672 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.845410 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:35:44.845418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:35:44.845444 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:35:44.845489 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:35:44.847722 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:44.848177 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:44.848184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:44.848191 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:44.848364 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:35:44.848602 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:35:44.848605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:35:44.848607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:35:44.849118 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:35:44.849319 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 04:35:44.849649 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 04:35:44.849904 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:44.851939 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.846411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:52417)
0809 04:35:44.846565 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.846581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.846667 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.846948 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.846951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.847090 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.847212 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.847218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.848131 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.848135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.848913 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.848925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.849457 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.849461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.849571 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.850017 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.850031 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.851801 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 04:35:44.852035 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.846539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:52417 (local address 127.19.92.62:38445)
0809 04:35:44.846657 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.846661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.846674 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:44.846750 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.846753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.846799 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.846874 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.847324 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.848024 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.849039 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.849341 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.849481 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.851934 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:52417: BlockingRecv error: recv got EOF from 127.19.92.3:52417 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:44 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:44.857074 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:44.852494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:54665)
0809 04:35:44.852622 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:44.852637 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:44.852713 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:44.853022 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:44.853025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:44.853174 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:44.853265 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.853271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.853943 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.853947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.854470 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:44.854477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.854870 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.854874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:44.854951 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.855378 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:44.855389 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:44.856948 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 04:35:44.857216 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:44.852562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:54665 (local address 127.19.92.62:38445)
0809 04:35:44.852685 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:44.852688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:44.852699 (+ 11us) server_negotiation.cc:407] Connection header received
0809 04:35:44.852793 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:44.852795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:44.852830 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:44.852896 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:44.853367 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.853846 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.854573 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:44.854778 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:44.854858 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:44.857100 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:54665: BlockingRecv error: recv got EOF from 127.19.92.3:54665 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 04:35:44.857342 920 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 04:35:45 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:45 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:45.475811 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:45.470496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:41321)
0809 04:35:45.470768 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:45.470784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:45.470887 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:45.471103 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:45.471105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:45.471283 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:45.471379 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:45.471384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.472166 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.472169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:45.472795 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:45.472806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.473404 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.473408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:45.473491 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:45.473939 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:45.473950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:45.475682 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
W20260809 04:35:45.476002 929 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:45.470603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:41321 (local address 127.19.92.62:38445)
0809 04:35:45.470885 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:45.470889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:45.470904 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:35:45.470955 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:45.470957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:45.471002 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:45.471078 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:45.471492 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.472046 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.472964 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.473279 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.473385 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:45.475858 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:41321: BlockingRecv error: recv got EOF from 127.19.92.1:41321 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":116,"threads_started":1}
Aug 09 04:35:45 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:45 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:45.679291 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:45.674065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:51761)
0809 04:35:45.674331 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:45.674346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:45.674454 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:45.674717 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:45.674720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:45.674914 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:45.675008 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:45.675014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.675753 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.675757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:45.676336 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:45.676344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.676844 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.676848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:45.676935 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:45.677409 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:45.677420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:45.679128 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260809 04:35:45.679371 929 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:45.674178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:51761 (local address 127.19.92.62:38445)
0809 04:35:45.674331 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:45.674336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:45.674412 (+ 76us) server_negotiation.cc:407] Connection header received
0809 04:35:45.674561 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:45.674565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:45.674614 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:45.674683 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:45.675119 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.675655 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.676466 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.676735 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.676818 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:45.679267 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:51761: BlockingRecv error: recv got EOF from 127.19.92.2:51761 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 04:35:45 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:45 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:45.863150 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:45.857938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:35705)
0809 04:35:45.858203 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:45.858216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:45.858312 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:45.858517 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:45.858519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:45.858695 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:45.858798 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:45.858803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.859516 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.859518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:45.860113 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:45.860123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.860591 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.860594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:45.860669 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:45.861208 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:45.861220 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:45.863019 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
W20260809 04:35:45.863276 929 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:45.858010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:35705 (local address 127.19.92.62:38445)
0809 04:35:45.858136 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:45.858139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:45.858274 (+ 135us) server_negotiation.cc:407] Connection header received
0809 04:35:45.858380 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:45.858382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:45.858428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:45.858514 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:45.858878 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.859391 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.860227 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:45.860449 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:45.860550 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:45.863166 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:35705: BlockingRecv error: recv got EOF from 127.19.92.3:35705 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 04:35:46 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:46 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:46.483265 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:46.476779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:45349)
0809 04:35:46.477088 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:46.477105 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:46.477277 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:46.477568 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:46.477571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:46.477766 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:46.477864 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:46.477872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.478953 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.478957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:46.479537 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:46.479546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.480256 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.480260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:46.480352 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:46.480904 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:46.480920 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:46.483101 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
W20260809 04:35:46.483749 936 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:46.476865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:45349 (local address 127.19.92.62:38445)
0809 04:35:46.477176 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:46.477180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:46.477198 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:35:46.477379 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:46.477382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:46.477437 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:46.477541 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:46.478004 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.478841 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.479651 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.480004 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.480104 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:46.483292 (+ 3188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:45349: BlockingRecv error: recv got EOF from 127.19.92.1:45349 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":60,"threads_started":1}
Aug 09 04:35:46 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:46 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:46.688846 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:46.680102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:54559)
0809 04:35:46.680407 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:46.680423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:46.680557 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:46.680813 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:46.680816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:46.681023 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:46.681125 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:46.681131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.681977 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.681980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:46.682629 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:46.682639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.683123 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.683127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:46.683207 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:46.686549 (+ 3342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:46.686565 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:46.688639 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
W20260809 04:35:46.688925 936 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:46.680182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:54559 (local address 127.19.92.62:38445)
0809 04:35:46.680332 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:46.680336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:46.680491 (+ 155us) server_negotiation.cc:407] Connection header received
0809 04:35:46.680656 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:46.680659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:46.680706 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:46.680774 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:46.681305 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.681856 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.682739 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.682982 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.683066 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:46.688815 (+ 5749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:54559: BlockingRecv error: recv got EOF from 127.19.92.2:54559 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 04:35:46 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:46 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:46.869961 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:46.864090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:35875)
0809 04:35:46.864399 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:46.864432 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:46.864551 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:46.864789 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:46.864791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:46.865006 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:46.865118 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:46.865124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.865945 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.865948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:46.866711 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:46.866725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.867273 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.867277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:46.867350 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:46.867878 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:46.867893 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:46.869814 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
W20260809 04:35:46.870066 936 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:46.864274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:35875 (local address 127.19.92.62:38445)
0809 04:35:46.864404 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:46.864408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:46.864502 (+ 94us) server_negotiation.cc:407] Connection header received
0809 04:35:46.864635 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:46.864638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:46.864684 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:46.864758 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:46.865251 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.865824 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.866839 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:46.867155 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:46.867250 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:46.869949 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:35875: BlockingRecv error: recv got EOF from 127.19.92.3:35875 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 04:35:47 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:47 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:47.490103 941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:47.484185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:41645)
0809 04:35:47.484506 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:47.484521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:47.484617 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:47.484849 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:47.484851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:47.485037 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:47.485210 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:47.485218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.486038 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.486041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:47.486640 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:47.486649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.487135 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.487138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:47.487220 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:47.487702 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:47.487714 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:47.489899 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260809 04:35:47.490163 942 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:47.484289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:41645 (local address 127.19.92.62:38445)
0809 04:35:47.484595 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:47.484598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:47.484611 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:47.484696 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:47.484698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:47.484742 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:47.484823 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:47.485353 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.485928 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.486759 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.487005 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.487095 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:47.490045 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:41645: BlockingRecv error: recv got EOF from 127.19.92.1:41645 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":127,"threads_started":1}
Aug 09 04:35:47 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:47 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:47.695222 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:47.689754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:39129)
0809 04:35:47.690042 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:47.690056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:47.690167 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:47.690414 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:47.690417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:47.690632 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:47.690729 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:47.690735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.691501 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.691505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:47.692077 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:47.692086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.692533 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.692537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:47.692623 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:47.693126 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:47.693137 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:47.695077 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
W20260809 04:35:47.695333 942 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:47.689853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:39129 (local address 127.19.92.62:38445)
0809 04:35:47.690006 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:47.690011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:47.690123 (+ 112us) server_negotiation.cc:407] Connection header received
0809 04:35:47.690258 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:47.690261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:47.690307 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:47.690372 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:47.690850 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.691387 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.692192 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.692428 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.692519 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:47.695228 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:39129: BlockingRecv error: recv got EOF from 127.19.92.2:39129 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 04:35:47 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:47 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:47.876123 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:47.870723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:48107)
0809 04:35:47.870976 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:47.870989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:47.871100 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:47.871311 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:47.871314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:47.871494 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:47.871598 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:47.871604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.872304 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.872306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:47.872893 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:47.872901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.873375 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.873379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:47.873451 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:47.874060 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:47.874076 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:47.875975 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260809 04:35:47.876214 942 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:47.870802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:48107 (local address 127.19.92.62:38445)
0809 04:35:47.870986 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:47.870990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:47.871063 (+ 73us) server_negotiation.cc:407] Connection header received
0809 04:35:47.871171 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:47.871173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:47.871220 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:47.871285 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:47.871689 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.872225 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.872994 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:47.873249 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:47.873328 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:47.876116 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:48107: BlockingRecv error: recv got EOF from 127.19.92.3:48107 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 09 04:35:48 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:48 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:48.496439 946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:48.490935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:38019)
0809 04:35:48.491240 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:48.491254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:48.491347 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:48.491696 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:48.491698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:48.491874 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:48.491974 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:48.491979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.492677 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.492680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:48.493283 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:48.493293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.493839 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.493843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:48.493911 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:48.494368 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:48.494379 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:48.496323 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
W20260809 04:35:48.496670 947 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:48.491055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:38019 (local address 127.19.92.62:38445)
0809 04:35:48.491305 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:48.491308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:48.491322 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:48.491520 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:48.491523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:48.491583 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:48.491667 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:48.492066 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.492601 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.493448 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.493736 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.493828 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:48.496488 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:38019: BlockingRecv error: recv got EOF from 127.19.92.1:38019 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":59,"threads_started":1}
Aug 09 04:35:48 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:48 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:48.701502 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:48.695970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:50731)
0809 04:35:48.696269 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:48.696284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:48.696401 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:48.696798 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:48.696801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:48.697026 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:48.697126 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:48.697133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.697965 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.697969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:48.698534 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:48.698543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.698975 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.698978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:48.699061 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:48.699527 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:48.699539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:48.701377 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":114,"threads_started":1}
W20260809 04:35:48.701682 947 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:48.696121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:50731 (local address 127.19.92.62:38445)
0809 04:35:48.696272 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:48.696276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:48.696353 (+ 77us) server_negotiation.cc:407] Connection header received
0809 04:35:48.696524 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:48.696528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:48.696649 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:48.696738 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:48.697287 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.697848 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.698647 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.698869 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.698956 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:48.701564 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:50731: BlockingRecv error: recv got EOF from 127.19.92.2:50731 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 04:35:48 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:48 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:48.882395 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:48.876909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:53017)
0809 04:35:48.877242 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:48.877258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:48.877377 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:48.877630 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:48.877634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:48.877845 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:48.877948 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:48.877955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.878703 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.878707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:48.879466 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:48.879478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.879909 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.879911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:48.879974 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:48.880515 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:48.880527 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:48.882279 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":113,"threads_started":1}
W20260809 04:35:48.882619 947 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:48.877073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:53017 (local address 127.19.92.62:38445)
0809 04:35:48.877243 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:48.877247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:48.877333 (+ 86us) server_negotiation.cc:407] Connection header received
0809 04:35:48.877465 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:48.877468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:48.877520 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:48.877588 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:48.878074 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.878597 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.879582 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:48.879819 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:48.879899 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:48.882406 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:53017: BlockingRecv error: recv got EOF from 127.19.92.3:53017 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 09 04:35:49 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:49 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:49.502753 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:49.497376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:42309)
0809 04:35:49.497655 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:49.497668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:49.497764 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:49.498034 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:49.498037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:49.498223 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:49.498324 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:49.498330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.499233 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.499236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:49.499823 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:49.499834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.500285 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.500289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:49.500377 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:49.500865 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:49.500877 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:49.502603 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
W20260809 04:35:49.502877 952 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:49.497484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:42309 (local address 127.19.92.62:38445)
0809 04:35:49.497734 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:49.497738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:49.497753 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:35:49.497847 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:49.497849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:49.497896 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:49.497993 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:49.498468 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.499093 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.499944 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.500175 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.500264 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:49.502759 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:42309: BlockingRecv error: recv got EOF from 127.19.92.1:42309 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":64,"threads_started":1}
Aug 09 04:35:49 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:49 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:49.707453 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:49.702369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:48595)
0809 04:35:49.702665 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:49.702678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:49.702779 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:49.702995 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:49.702997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:49.703190 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:49.703286 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:49.703290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.703995 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.703998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:49.704601 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:49.704610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.705032 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.705035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:49.705108 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:49.705626 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:49.705637 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:49.707327 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
W20260809 04:35:49.707585 952 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:49.702531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:48595 (local address 127.19.92.62:38445)
0809 04:35:49.702671 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:49.702674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:49.702731 (+ 57us) server_negotiation.cc:407] Connection header received
0809 04:35:49.702860 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:49.702862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:49.702906 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:49.702963 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:49.703371 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.703909 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.704690 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.704907 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.704977 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:49.707470 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:48595: BlockingRecv error: recv got EOF from 127.19.92.2:48595 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 04:35:49 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:49 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:49.888940 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:49.883247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:37145)
0809 04:35:49.883561 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:49.883573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:49.883683 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:49.883927 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:49.883930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:49.884155 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:49.884255 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:49.884262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.885087 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.885090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:49.885711 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:49.885721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.886248 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.886252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:49.886344 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:49.886889 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:49.886901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:49.888816 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":115,"threads_started":1}
W20260809 04:35:49.889087 952 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:49.883343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:37145 (local address 127.19.92.62:38445)
0809 04:35:49.883477 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:49.883482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:49.883643 (+ 161us) server_negotiation.cc:407] Connection header received
0809 04:35:49.883775 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:49.883778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:49.883827 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:49.883892 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:49.884375 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.884969 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.885827 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:49.886117 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:49.886221 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:49.888972 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:37145: BlockingRecv error: recv got EOF from 127.19.92.3:37145 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 09 04:35:50 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:50 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:50.508693 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:50.503526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:55823)
0809 04:35:50.503807 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:50.503820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:50.503931 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:50.504176 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:50.504177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:50.504364 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:50.504479 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:50.504484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.505227 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.505231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:50.505817 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:50.505825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.506234 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.506236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:50.506294 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:50.506776 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:50.506787 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:50.508536 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
W20260809 04:35:50.508808 957 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:50.503587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:55823 (local address 127.19.92.62:38445)
0809 04:35:50.503819 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:50.503822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:50.503887 (+ 65us) server_negotiation.cc:407] Connection header received
0809 04:35:50.504021 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:50.504024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:50.504074 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:50.504149 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:50.504568 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.505111 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.505909 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.506148 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.506225 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:50.508688 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:55823: BlockingRecv error: recv got EOF from 127.19.92.1:55823 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":59,"threads_started":1}
Aug 09 04:35:50 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:50 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:50.713780 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:50.708309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:40275)
0809 04:35:50.708577 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:50.708590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:50.708697 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:50.708975 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:50.708977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:50.709231 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:50.709375 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:50.709383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.710322 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.710326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:50.710971 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:50.710981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.711406 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.711408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:50.711495 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:50.711943 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:50.711954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:50.713663 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
W20260809 04:35:50.713894 957 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:50.708411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:40275 (local address 127.19.92.62:38445)
0809 04:35:50.708568 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:50.708572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:50.708655 (+ 83us) server_negotiation.cc:407] Connection header received
0809 04:35:50.708771 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:50.708773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:50.708820 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:50.708923 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:50.709502 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.710204 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.711076 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.711321 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.711400 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:50.713795 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:40275: BlockingRecv error: recv got EOF from 127.19.92.2:40275 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 04:35:50 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:50 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:50.895433 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:50.889829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:59041)
0809 04:35:50.890119 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:50.890133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:50.890256 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:50.890520 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:50.890523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:50.890745 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:50.890848 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:50.890854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.891631 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.891635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:50.892211 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:50.892219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.892674 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.892678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:50.892762 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:50.893295 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:50.893307 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:50.895276 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":95,"threads_started":1}
W20260809 04:35:50.895529 957 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:50.889935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:59041 (local address 127.19.92.62:38445)
0809 04:35:50.890111 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:50.890116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:50.890208 (+ 92us) server_negotiation.cc:407] Connection header received
0809 04:35:50.890349 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:50.890352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:50.890405 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:50.890505 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:50.890975 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.891520 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.892328 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:50.892565 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:50.892654 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:50.895418 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:59041: BlockingRecv error: recv got EOF from 127.19.92.3:59041 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 09 04:35:51 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:51 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:51.515133 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:51.509510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:56579)
0809 04:35:51.509824 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:51.509838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:51.509939 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:51.510204 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:51.510207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:51.510428 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:51.510532 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:51.510538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.511436 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.511439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:51.512198 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:51.512208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.512659 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.512662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:51.512722 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:51.513216 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:51.513230 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:51.515017 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":69,"threads_started":1}
W20260809 04:35:51.515287 964 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:51.509592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:56579 (local address 127.19.92.62:38445)
0809 04:35:51.509843 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:51.509846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:51.509900 (+ 54us) server_negotiation.cc:407] Connection header received
0809 04:35:51.510010 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:51.510012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:51.510088 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:51.510174 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:51.510632 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.511315 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.512305 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.512557 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.512636 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:51.515151 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:56579: BlockingRecv error: recv got EOF from 127.19.92.1:56579 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
Aug 09 04:35:51 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:51 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:51.723225 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:51.714587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:35599)
0809 04:35:51.714912 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:51.714925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:51.715067 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:51.715281 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:51.715283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:51.715476 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:51.715570 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:51.715575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.716354 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.716357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:51.717005 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:51.717016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.717537 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.717540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:51.717610 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:51.720867 (+ 3257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:51.720881 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:51.723070 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
W20260809 04:35:51.723326 964 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:51.714674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:35599 (local address 127.19.92.62:38445)
0809 04:35:51.714849 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:51.714853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:51.715000 (+ 147us) server_negotiation.cc:407] Connection header received
0809 04:35:51.715141 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:51.715143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:51.715189 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:51.715256 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:51.715656 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.716243 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.717172 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.717402 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.717519 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:51.723225 (+ 5706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:35599: BlockingRecv error: recv got EOF from 127.19.92.2:35599 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 09 04:35:51 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:51 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:51.901808 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:51.896352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:56423)
0809 04:35:51.896686 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:51.896700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:51.896804 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:51.897037 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:51.897039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:51.897247 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:51.897349 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:51.897354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.898177 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.898180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:51.898898 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:51.898910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.899360 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.899363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:51.899431 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:51.899922 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:51.899934 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:51.901689 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
W20260809 04:35:51.901975 964 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:51.896461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:56423 (local address 127.19.92.62:38445)
0809 04:35:51.896600 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:51.896603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:51.896768 (+ 165us) server_negotiation.cc:407] Connection header received
0809 04:35:51.896877 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:51.896879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:51.896928 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:51.897000 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:51.897482 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.898067 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.899035 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:51.899256 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:51.899332 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:51.901856 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:56423: BlockingRecv error: recv got EOF from 127.19.92.3:56423 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 04:35:52 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:52 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:52.521023 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:52.515941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:44277)
0809 04:35:52.516203 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:52.516217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:52.516308 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:52.516528 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:52.516530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:52.516710 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:52.516809 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:52.516815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.517683 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.517687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:52.518261 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:52.518271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.518717 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.518721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:52.518807 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:52.519241 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:52.519252 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:52.520862 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
W20260809 04:35:52.521121 971 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:52.516091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:44277 (local address 127.19.92.62:38445)
0809 04:35:52.516307 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:52.516310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:52.516323 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:52.516377 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:52.516379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:52.516421 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:52.516488 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:52.516928 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.517581 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.518379 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.518612 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.518696 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:52.521002 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:44277: BlockingRecv error: recv got EOF from 127.19.92.1:44277 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
Aug 09 04:35:52 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:52 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:52.729707 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:52.724155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:32905)
0809 04:35:52.724453 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:52.724466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:52.724577 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:52.724791 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:52.724793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:52.724971 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:52.725092 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:52.725098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.726037 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.726040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:52.726618 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:52.726627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.727037 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.727039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:52.727101 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:52.727579 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:52.727590 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:52.729551 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260809 04:35:52.729800 971 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:52.724260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:32905 (local address 127.19.92.62:38445)
0809 04:35:52.724436 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:52.724441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:52.724526 (+ 85us) server_negotiation.cc:407] Connection header received
0809 04:35:52.724658 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:52.724660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:52.724706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:52.724763 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:52.725221 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.725901 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.726722 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.726954 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.727025 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:52.729700 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:32905: BlockingRecv error: recv got EOF from 127.19.92.2:32905 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 04:35:52 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:52 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:52.908111 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:52.902678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:51757)
0809 04:35:52.903008 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:52.903025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:52.903150 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:52.903440 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:52.903444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:52.903693 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:52.903823 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:52.903831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.904634 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.904636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:52.905237 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:52.905245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.905684 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.905687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:52.905754 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:52.906224 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:52.906235 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:52.907994 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
W20260809 04:35:52.908260 971 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:52.902802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:51757 (local address 127.19.92.62:38445)
0809 04:35:52.902985 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:52.902989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:52.903107 (+ 118us) server_negotiation.cc:407] Connection header received
0809 04:35:52.903248 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:52.903251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:52.903314 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:52.903391 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:52.903937 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.904538 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.905339 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:52.905587 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:52.905665 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:52.908136 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:51757: BlockingRecv error: recv got EOF from 127.19.92.3:51757 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 09 04:35:53 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:53 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:53.527305 976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:53.521890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:56023)
0809 04:35:53.522158 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:53.522174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:53.522282 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:53.522577 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:53.522579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:53.522814 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:53.522940 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:53.522949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.523785 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.523788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:53.524349 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:53.524359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.524734 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.524737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:53.524804 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:53.525309 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:53.525320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:53.527181 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
W20260809 04:35:53.527518 977 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:53.521983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:56023 (local address 127.19.92.62:38445)
0809 04:35:53.522249 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:53.522253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:53.522267 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:35:53.522378 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:53.522381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:53.522430 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:53.522510 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:53.523065 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.523678 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.524447 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.524659 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.524723 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:53.527371 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:56023: BlockingRecv error: recv got EOF from 127.19.92.1:56023 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
Aug 09 04:35:53 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:53 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:53.736392 981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:53.730747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:57779)
0809 04:35:53.731228 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:53.731244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:53.731362 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:53.731693 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:53.731696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:53.731916 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:53.732055 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:53.732063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.732878 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.732881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:53.733503 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:53.733511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.733906 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.733908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:53.733980 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:53.734471 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:53.734484 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:53.736275 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":106,"threads_started":1}
W20260809 04:35:53.736644 977 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:53.730911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:57779 (local address 127.19.92.62:38445)
0809 04:35:53.731060 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:53.731063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:53.731313 (+ 250us) server_negotiation.cc:407] Connection header received
0809 04:35:53.731513 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:53.731516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:53.731575 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:53.731638 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:53.732156 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.732776 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.733596 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.733812 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.733888 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:53.736417 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:57779: BlockingRecv error: recv got EOF from 127.19.92.2:57779 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 04:35:53 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:53 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:53.914225 982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:53.908912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:56217)
0809 04:35:53.909183 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:53.909196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:53.909294 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:53.909512 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:53.909514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:53.909692 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:53.909794 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:53.909799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.910669 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.910672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:53.911271 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:53.911281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.911685 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.911687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:53.911747 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:53.912214 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:53.912224 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:53.914109 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
W20260809 04:35:53.914379 977 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:53.908976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:56217 (local address 127.19.92.62:38445)
0809 04:35:53.909123 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:53.909128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:53.909255 (+ 127us) server_negotiation.cc:407] Connection header received
0809 04:35:53.909371 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:53.909373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:53.909422 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:53.909487 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:53.909953 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.910563 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.911370 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:53.911601 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:53.911679 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:53.914263 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:56217: BlockingRecv error: recv got EOF from 127.19.92.3:56217 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 04:35:54 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:54 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:54.533955 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:54.528181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:48855)
0809 04:35:54.528522 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:54.528539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:54.528648 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:54.528930 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:54.528932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:54.529121 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:54.529266 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:54.529272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.530129 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.530132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:54.530746 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:54.530755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.531260 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.531263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:54.531336 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:54.531919 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:54.531931 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:54.533826 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
W20260809 04:35:54.534128 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:54.528277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:48855 (local address 127.19.92.62:38445)
0809 04:35:54.528545 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:54.528549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:54.528614 (+ 65us) server_negotiation.cc:407] Connection header received
0809 04:35:54.528725 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:54.528727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:54.528770 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:54.528848 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:54.529421 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.529989 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.530851 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.531134 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.531212 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:54.533976 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:48855: BlockingRecv error: recv got EOF from 127.19.92.1:48855 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":65,"threads_started":1}
Aug 09 04:35:54 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:54 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:54.742650 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:54.737359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:45557)
0809 04:35:54.737658 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:54.737681 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:54.737785 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:54.738051 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:54.738054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:54.738256 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:54.738348 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:54.738354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.739111 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.739115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:54.739683 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:54.739691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.740125 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.740128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:54.740210 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:54.740717 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:54.740730 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:54.742460 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
W20260809 04:35:54.742709 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:54.737420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:45557 (local address 127.19.92.62:38445)
0809 04:35:54.737592 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:54.737597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:54.737739 (+ 142us) server_negotiation.cc:407] Connection header received
0809 04:35:54.737878 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:54.737881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:54.737929 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:54.738001 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:54.738465 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.739009 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.739793 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.740026 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.740110 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:54.742604 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:45557: BlockingRecv error: recv got EOF from 127.19.92.2:45557 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 04:35:54 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:54 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:54.920554 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:54.915102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:43749)
0809 04:35:54.915376 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:54.915389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:54.915487 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:54.915694 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:54.915696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:54.915935 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:54.916060 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:54.916067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.916818 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.916821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:54.917517 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:54.917528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.917983 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.917987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:54.918061 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:54.918570 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:54.918582 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:54.920380 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
W20260809 04:35:54.920665 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:54.915193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:43749 (local address 127.19.92.62:38445)
0809 04:35:54.915324 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:54.915328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:54.915443 (+ 115us) server_negotiation.cc:407] Connection header received
0809 04:35:54.915556 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:54.915558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:54.915603 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:54.915666 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:54.916166 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.916719 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.917626 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:54.917857 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:54.917941 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:54.920555 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:43749: BlockingRecv error: recv got EOF from 127.19.92.3:43749 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 04:35:55 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:55 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:55.540086 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:55.534794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:47163)
0809 04:35:55.535077 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:55.535090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:55.535183 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:55.535419 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:55.535421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:55.535601 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:55.535703 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:55.535708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.536456 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.536458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:55.537015 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:55.537022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.537457 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.537460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:55.537532 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:55.538051 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:55.538064 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:55.539887 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
W20260809 04:35:55.540185 992 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:55.534898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:47163 (local address 127.19.92.62:38445)
0809 04:35:55.535120 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:55.535124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:55.535137 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:55.535264 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:55.535267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:55.535313 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:55.535393 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:55.535822 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.536372 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.537093 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.537361 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.537433 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:55.540063 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:47163: BlockingRecv error: recv got EOF from 127.19.92.1:47163 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
Aug 09 04:35:55 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:55 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:55.748714 994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:55.743444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:35881)
0809 04:35:55.743732 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:55.743745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:55.743854 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:55.744117 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:55.744121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:55.744325 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:55.744432 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:55.744438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.745239 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.745243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:55.745813 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:55.745821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.746254 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.746257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:55.746341 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:55.746835 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:55.746849 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:55.748566 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
W20260809 04:35:55.748806 992 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:55.743526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:35881 (local address 127.19.92.62:38445)
0809 04:35:55.743675 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:55.743680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:55.743812 (+ 132us) server_negotiation.cc:407] Connection header received
0809 04:35:55.743945 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:55.743963 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:55.744010 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:55.744075 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:55.744567 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.745107 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.745923 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.746154 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.746240 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:55.748706 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:35881: BlockingRecv error: recv got EOF from 127.19.92.2:35881 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 04:35:55 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:55 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:55.926889 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:55.921429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:53433)
0809 04:35:55.921724 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:55.921739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:55.921854 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:55.922110 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:55.922114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:55.922312 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:55.922412 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:55.922418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.923176 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.923180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:55.923746 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:55.923754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.924188 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.924191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:55.924272 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:55.924825 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:55.924838 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:55.926733 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":112,"threads_started":1}
W20260809 04:35:55.926985 992 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:55.921552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:53433 (local address 127.19.92.62:38445)
0809 04:35:55.921720 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:55.921724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:55.921810 (+ 86us) server_negotiation.cc:407] Connection header received
0809 04:35:55.921940 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:55.921944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:55.922000 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:55.922072 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:55.922527 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.923073 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.923856 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:55.924085 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:55.924172 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:55.926880 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:53433: BlockingRecv error: recv got EOF from 127.19.92.3:53433 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 04:35:56 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:56 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:56.547822 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:56.540997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:38703)
0809 04:35:56.541268 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:56.541283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:56.541387 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:56.541863 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:56.541866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:56.542103 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:56.542233 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:56.542242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.543446 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.543451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:56.544326 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:56.544340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.544821 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.544825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:56.544919 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:56.545562 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:56.545577 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:56.547658 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":65,"threads_started":1}
W20260809 04:35:56.547904 1000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:56.541336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:38703 (local address 127.19.92.62:38445)
0809 04:35:56.541597 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:56.541601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:56.541617 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:35:56.541663 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:56.541666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:56.541715 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:56.541815 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:56.542548 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.543325 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.544459 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.544709 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.544785 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:56.547791 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:38703: BlockingRecv error: recv got EOF from 127.19.92.1:38703 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
Aug 09 04:35:56 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:56 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:56.758235 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:56.749674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:54039)
0809 04:35:56.749958 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:56.749972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:56.750076 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:56.750353 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:56.750356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:56.750543 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:56.750644 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:56.750650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.751538 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.751541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:56.752310 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:56.752321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.752756 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.752759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:56.752824 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:56.756258 (+ 3434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:56.756270 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:56.758112 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
W20260809 04:35:56.758368 1000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:56.749794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:54039 (local address 127.19.92.62:38445)
0809 04:35:56.749966 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:56.749969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:56.750029 (+ 60us) server_negotiation.cc:407] Connection header received
0809 04:35:56.750164 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:56.750166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:56.750212 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:56.750273 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:56.750743 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.751412 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.752423 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.752649 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.752729 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:56.758259 (+ 5530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:54039: BlockingRecv error: recv got EOF from 127.19.92.2:54039 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 09 04:35:56 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:56 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:56.933092 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:56.927739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:50821)
0809 04:35:56.928038 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:56.928052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:56.928162 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:56.928422 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:56.928425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:56.928630 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:56.928741 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:56.928747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.929473 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.929476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:56.930056 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:56.930063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.930445 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.930447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:56.930507 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:56.931088 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:56.931101 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:56.932925 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
W20260809 04:35:56.933205 1000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:56.927888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:50821 (local address 127.19.92.62:38445)
0809 04:35:56.928040 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:56.928044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:56.928118 (+ 74us) server_negotiation.cc:407] Connection header received
0809 04:35:56.928234 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:56.928236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:56.928283 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:56.928348 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:56.928838 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.929383 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.930147 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:56.930362 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:56.930431 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:56.933074 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:50821: BlockingRecv error: recv got EOF from 127.19.92.3:50821 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 04:35:57 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:57 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:57.555217 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:57.550107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:33327)
0809 04:35:57.550367 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:57.550380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:57.550473 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:57.550684 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:57.550686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:57.550867 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:57.550985 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:57.550990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.551689 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.551691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:57.552256 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:57.552263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.552686 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.552689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:57.552764 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:57.553312 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:57.553324 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:57.555082 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260809 04:35:57.555413 1006 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:57.550226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:33327 (local address 127.19.92.62:38445)
0809 04:35:57.550456 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:57.550459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:57.550472 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:57.550541 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:57.550543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:57.550586 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:57.550659 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:57.551078 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.551611 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.552337 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.552570 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.552643 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:57.555262 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:33327: BlockingRecv error: recv got EOF from 127.19.92.1:33327 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
Aug 09 04:35:57 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:57 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:57.764183 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:57.759010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:39009)
0809 04:35:57.759290 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:57.759306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:57.759423 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:57.759668 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:57.759671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:57.759874 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:57.759966 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:57.759972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.760716 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.760720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:57.761316 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:57.761325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.761751 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.761754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:57.761832 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:57.762317 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:57.762329 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:57.764030 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
W20260809 04:35:57.764281 1006 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:57.759171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:39009 (local address 127.19.92.62:38445)
0809 04:35:57.759297 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:57.759302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:57.759372 (+ 70us) server_negotiation.cc:407] Connection header received
0809 04:35:57.759507 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:57.759510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:57.759561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:57.759627 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:57.760085 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.760611 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.761427 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.761653 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.761737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:57.764178 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:39009: BlockingRecv error: recv got EOF from 127.19.92.2:39009 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 04:35:57 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:57 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:57.939018 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:57.933891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:44447)
0809 04:35:57.934152 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:57.934164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:57.934265 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:57.934480 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:57.934483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:57.934687 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:57.934789 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:57.934795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.935509 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.935511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:57.936080 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:57.936087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.936527 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.936529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:57.936589 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:57.937110 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:57.937123 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:57.938894 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
W20260809 04:35:57.939160 1006 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:57.933976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:44447 (local address 127.19.92.62:38445)
0809 04:35:57.934100 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:57.934104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:57.934216 (+ 112us) server_negotiation.cc:407] Connection header received
0809 04:35:57.934344 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:57.934346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:57.934391 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:57.934454 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:57.934873 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.935390 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.936178 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:57.936403 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:57.936504 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:57.939037 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:44447: BlockingRecv error: recv got EOF from 127.19.92.3:44447 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 04:35:58 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:58 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:58.561451 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:58.556050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:36679)
0809 04:35:58.556323 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:58.556336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:58.556429 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:58.556682 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:58.556684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:58.556866 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:58.556981 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:58.556987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.557760 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.557763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:58.558368 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:58.558376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.558865 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.558869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:58.558980 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:58.559479 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:58.559491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:58.561282 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
W20260809 04:35:58.561554 1011 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:58.556139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:36679 (local address 127.19.92.62:38445)
0809 04:35:58.556420 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:58.556423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:58.556438 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:35:58.556531 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:58.556533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:58.556578 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:58.556656 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:58.557070 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.557659 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.558467 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.558745 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.558831 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:58.561423 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:36679: BlockingRecv error: recv got EOF from 127.19.92.1:36679 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
Aug 09 04:35:58 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:58 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:58.770763 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:58.765118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:56747)
0809 04:35:58.765427 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:58.765469 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:58.765572 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:58.765847 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:58.765850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:58.766051 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:58.766153 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:58.766159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.766916 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.766918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:58.767685 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:58.767695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.768235 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.768238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:58.768302 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:58.768803 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:58.768815 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:58.770623 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
W20260809 04:35:58.770881 1011 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:58.765237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:56747 (local address 127.19.92.62:38445)
0809 04:35:58.765361 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:58.765364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:58.765537 (+ 173us) server_negotiation.cc:407] Connection header received
0809 04:35:58.765644 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:58.765646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:58.765692 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:58.765765 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:58.766288 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.766817 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.767879 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.768132 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.768218 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:58.770774 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:56747: BlockingRecv error: recv got EOF from 127.19.92.2:56747 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 04:35:58 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:58 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.3@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:58.945331 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:58.939815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.3:57321)
0809 04:35:58.940097 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:58.940111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:58.940215 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:58.940451 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:58.940453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:58.940641 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:58.940743 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:58.940748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.941744 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.941748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:58.942397 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:58.942409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.942823 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.942826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:58.942890 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:58.943396 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:58.943409 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:58.945203 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
W20260809 04:35:58.945410 1011 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:58.939917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:57321 (local address 127.19.92.62:38445)
0809 04:35:58.940102 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:58.940105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:58.940176 (+ 71us) server_negotiation.cc:407] Connection header received
0809 04:35:58.940288 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:58.940290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:58.940343 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:58.940430 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:58.940844 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.941623 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.942506 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:58.942733 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:58.942808 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:58.945314 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.3:57321: BlockingRecv error: recv got EOF from 127.19.92.3:57321 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 04:35:59 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:59 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.1@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:59.568148 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:59.562357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.1:38233)
0809 04:35:59.562657 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:59.562672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:59.562760 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:59.562975 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:59.562976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:59.563271 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:59.563414 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:59.563421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:59.564256 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:59.564259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:59.564875 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:59.564883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:59.565317 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:59.565320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:59.565383 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:59.565933 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:59.565946 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:59.568000 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
W20260809 04:35:59.568293 1019 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:59.562484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:38233 (local address 127.19.92.62:38445)
0809 04:35:59.562741 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:59.562745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:59.562758 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:35:59.562827 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:59.562829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:59.562870 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:59.562956 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:59.563609 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:59.564162 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:59.564983 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:59.565235 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:59.565312 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:59.568161 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.1:38233: BlockingRecv error: recv got EOF from 127.19.92.1:38233 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
Aug 09 04:35:59 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:35:59 dist-test-slave-rmhg krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.92.2@KRBTEST.COM for oryx/127.19.92.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:35:59.777477 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:35:59.771749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:38445 (local address 127.19.92.2:44121)
0809 04:35:59.772145 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0809 04:35:59.772160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:35:59.772293 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:35:59.772565 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:35:59.772567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:35:59.772791 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:35:59.772888 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:59.772894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:59.773731 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:59.773735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:59.774337 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:35:59.774346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:35:59.774799 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:35:59.774803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:35:59.774885 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:59.775387 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:35:59.775400 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:35:59.777315 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.92.62:38445: Server oryx/127.19.92.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
W20260809 04:35:59.777688 1019 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:35:59.771913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:44121 (local address 127.19.92.62:38445)
0809 04:35:59.772120 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 04:35:59.772124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:35:59.772252 (+ 128us) server_negotiation.cc:407] Connection header received
0809 04:35:59.772393 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:35:59.772396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:35:59.772450 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:35:59.772523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:35:59.773023 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:59.773622 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:59.774457 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:35:59.774694 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:35:59.774785 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:35:59.777574 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.92.2:44121: BlockingRecv error: recv got EOF from 127.19.92.2:44121 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 04:35:59.846632 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 32745
I20260809 04:35:59.862182 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 512
I20260809 04:35:59.871758 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 724
I20260809 04:35:59.879892 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 32636
2026-08-09T04:35:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18191 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T04:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:35:59Z Disabled control of system clock
I20260809 04:35:59.918184 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:44117
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:44117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 04:36:00.027454 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:00.027707 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:00.027740 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:00.031586 1024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:36:00.031639 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:00.031654 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:00.031669 1024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:36:00.031682 1024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:36:00.036187 1024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:44117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:44117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:00.037143 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:00.038455 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:36:00.046706 1024 server_base.cc:1034] running on GCE node
W20260809 04:36:00.046741 1038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.046671 1032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.046878 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:00.047394 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:00.048453 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:00.049647 1024 hybrid_clock.cc:648] HybridClock initialized: now 1786250160049627 us; error 30 us; skew 500 ppm
I20260809 04:36:00.051541 1024 webserver.cc:460] Webserver started at http://127.19.92.62:33255/ using document root <none> and password file <none>
I20260809 04:36:00.052098 1024 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:00.052157 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:00.052331 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:00.054471 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "860082d88c5d4f608902f1cb88a31ae5"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "9fe7f0b551445b1af63f1ec064f75234"
server_key_iv: "93a40cd6ab4b8992b7b07f2d628f02ab"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.054986 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "860082d88c5d4f608902f1cb88a31ae5"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "9fe7f0b551445b1af63f1ec064f75234"
server_key_iv: "93a40cd6ab4b8992b7b07f2d628f02ab"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.058557 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:36:00.060878 1048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.062161 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:36:00.062278 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "860082d88c5d4f608902f1cb88a31ae5"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "9fe7f0b551445b1af63f1ec064f75234"
server_key_iv: "93a40cd6ab4b8992b7b07f2d628f02ab"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.062391 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:00.072243 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:00.072853 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:00.073037 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:00.079562 1024 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:44117
I20260809 04:36:00.079576 1135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:44117 every 8 connection(s)
I20260809 04:36:00.080449 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:36:00.083513 1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:00.083850 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 1024
I20260809 04:36:00.083959 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:36:00.084285 19824 external_mini_cluster.cc:1428] Setting key b5cdda9f7b6e7130dc1534ea4edd781e
I20260809 04:36:00.089573 1136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5: Bootstrap starting.
I20260809 04:36:00.092149 1136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:00.092968 1136 log.cc:824] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:00.093730 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:00.085721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.19.92.62:44117)
0809 04:36:00.086302 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:00.086317 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:00.086351 (+ 34us) server_negotiation.cc:407] Connection header received
0809 04:36:00.087005 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:00.087023 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:00.087405 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:00.087710 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:00.088040 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.088991 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.090226 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.090725 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.090852 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:00.091725 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:00.091731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:00.091745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:36:00.092138 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:00.092195 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:36:00.092222 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:00.092321 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:00.092499 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 04:36:00.092786 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260809 04:36:00.095158 1136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5: No bootstrap required, opened a new log
I20260809 04:36:00.098670 1136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860082d88c5d4f608902f1cb88a31ae5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44117 } }
I20260809 04:36:00.098884 1136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:00.098940 1136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860082d88c5d4f608902f1cb88a31ae5, State: Initialized, Role: FOLLOWER
I20260809 04:36:00.099328 1136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860082d88c5d4f608902f1cb88a31ae5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44117 } }
I20260809 04:36:00.099453 1136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:36:00.099515 1136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:36:00.099613 1136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:00.100493 1136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860082d88c5d4f608902f1cb88a31ae5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44117 } }
I20260809 04:36:00.100840 1136 leader_election.cc:302] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 860082d88c5d4f608902f1cb88a31ae5; no voters:
I20260809 04:36:00.101192 1136 leader_election.cc:288] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:36:00.101277 1146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:36:00.101498 1146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [term 1 LEADER]: Becoming Leader. State: Replica: 860082d88c5d4f608902f1cb88a31ae5, State: Running, Role: LEADER
I20260809 04:36:00.101800 1146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860082d88c5d4f608902f1cb88a31ae5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44117 } }
I20260809 04:36:00.102317 1136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:36:00.103330 1147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "860082d88c5d4f608902f1cb88a31ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860082d88c5d4f608902f1cb88a31ae5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44117 } } }
I20260809 04:36:00.103521 1147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:36:00.103487 1148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 860082d88c5d4f608902f1cb88a31ae5. Latest consensus state: current_term: 1 leader_uuid: "860082d88c5d4f608902f1cb88a31ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860082d88c5d4f608902f1cb88a31ae5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 44117 } } }
I20260809 04:36:00.103569 1148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:36:00.103976 1156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:36:00.107290 1156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:36:00.112218 1156 catalog_manager.cc:1347] Generated new cluster ID: a135b8b933cc4900b459e797ea627376
I20260809 04:36:00.112334 1156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:36:00.134824 1156 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:36:00.135808 1156 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:36:00.143229 1156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 860082d88c5d4f608902f1cb88a31ae5: Generated new TSK 0
I20260809 04:36:00.143940 1156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:36:00.153331 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:44117
--builtin_ntp_servers=127.19.92.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 04:36:00.272595 1172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:00.272851 1172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:00.272888 1172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:00.276878 1172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:00.276937 1172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:00.276997 1172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:36:00.281796 1172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:44117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:00.282855 1172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:00.284238 1172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:00.293741 1181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.293778 1179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.293778 1183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:00.293973 1172 server_base.cc:1034] running on GCE node
I20260809 04:36:00.294464 1172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:00.295068 1172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:00.296219 1172 hybrid_clock.cc:648] HybridClock initialized: now 1786250160296190 us; error 41 us; skew 500 ppm
I20260809 04:36:00.298010 1172 webserver.cc:460] Webserver started at http://127.19.92.1:45171/ using document root <none> and password file <none>
I20260809 04:36:00.298565 1172 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:00.298629 1172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:00.298817 1172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:00.300866 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "e388a4f49e214f37aa8cacadaa11eb65"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "5287cf4e33746b6c36ede92d2d68be8c"
server_key_iv: "ccc452637414bb0fce7024de6050fb09"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.301440 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "e388a4f49e214f37aa8cacadaa11eb65"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "5287cf4e33746b6c36ede92d2d68be8c"
server_key_iv: "ccc452637414bb0fce7024de6050fb09"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.305343 1172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:36:00.307770 1194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.309116 1172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:36:00.309253 1172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "e388a4f49e214f37aa8cacadaa11eb65"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "5287cf4e33746b6c36ede92d2d68be8c"
server_key_iv: "ccc452637414bb0fce7024de6050fb09"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.309358 1172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:00.325132 1172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:00.325845 1172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:00.326030 1172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:00.326607 1172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:00.327566 1172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:00.327615 1172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.327666 1172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:00.327695 1172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.336067 1172 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:37329
I20260809 04:36:00.336126 1374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:37329 every 8 connection(s)
I20260809 04:36:00.337019 1172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:36:00.341557 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 1172
I20260809 04:36:00.341667 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:36:00.341938 19824 external_mini_cluster.cc:1428] Setting key 78ade564195e41461cc7c307074294a6
I20260809 04:36:00.343971 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:44117
--builtin_ntp_servers=127.19.92.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 04:36:00.347049 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:00.338866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:48375 (local address 127.19.92.62:44117)
0809 04:36:00.339010 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:00.339013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:00.340076 (+ 1063us) server_negotiation.cc:407] Connection header received
0809 04:36:00.341486 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:00.341489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:00.341553 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:00.341626 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:00.342741 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.343451 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.345022 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.345279 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.345359 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:00.346529 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:00.346533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:00.346536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:36:00.346550 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:00.346572 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:36:00.346580 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:00.346638 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:00.346827 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 04:36:00.346895 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:36:00.348088 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:00.339263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:44117 (local address 127.19.92.1:48375)
0809 04:36:00.339907 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:00.339953 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:00.341260 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:00.341822 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:00.341829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:00.341917 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:00.342574 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:00.342594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.343587 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.343591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:00.344905 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:00.344918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.345431 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.345436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:00.345634 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:00.346262 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:36:00.346264 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:00.346305 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:36:00.346371 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:00.346690 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:00.346699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:00.346705 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:36:00.346813 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 04:36:00.347110 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260809 04:36:00.349248 1376 heartbeater.cc:347] Connected to a master server at 127.19.92.62:44117
I20260809 04:36:00.349514 1376 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:00.350132 1376 heartbeater.cc:511] Master 127.19.92.62:44117 requested a full tablet report, sending...
I20260809 04:36:00.351629 1076 ts_manager.cc:194] Registered new tserver with Master: e388a4f49e214f37aa8cacadaa11eb65 (127.19.92.1:37329)
I20260809 04:36:00.352751 1076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.1:48375
W20260809 04:36:00.462409 1379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:00.462700 1379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:00.462756 1379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:00.466965 1379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:00.467031 1379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:00.467105 1379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:36:00.472035 1379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:44117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:00.473222 1379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:00.474651 1379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:00.484386 1392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.484390 1387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.484436 1388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:00.484483 1379 server_base.cc:1034] running on GCE node
I20260809 04:36:00.485102 1379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:00.485802 1379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:00.486974 1379 hybrid_clock.cc:648] HybridClock initialized: now 1786250160486931 us; error 53 us; skew 500 ppm
I20260809 04:36:00.488852 1379 webserver.cc:460] Webserver started at http://127.19.92.2:35703/ using document root <none> and password file <none>
I20260809 04:36:00.489502 1379 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:00.489571 1379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:00.489801 1379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:00.492026 1379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "5f75b63d52494b3180a7902b4a4555ad"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "33c7c72d1d92892aa5769ad732e5f2f9"
server_key_iv: "6760dfedb598de17c3545bf47e29d3e2"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.492653 1379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "5f75b63d52494b3180a7902b4a4555ad"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "33c7c72d1d92892aa5769ad732e5f2f9"
server_key_iv: "6760dfedb598de17c3545bf47e29d3e2"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.496567 1379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260809 04:36:00.499301 1406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.500584 1379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:36:00.500702 1379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "5f75b63d52494b3180a7902b4a4555ad"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "33c7c72d1d92892aa5769ad732e5f2f9"
server_key_iv: "6760dfedb598de17c3545bf47e29d3e2"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.500811 1379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:00.513015 1379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:00.513738 1379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:00.513913 1379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:00.514485 1379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:00.515431 1379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:00.515478 1379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.515532 1379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:00.515552 1379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.523854 1379 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:36423
I20260809 04:36:00.523875 1585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:36423 every 8 connection(s)
I20260809 04:36:00.524783 1379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:36:00.532204 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 1379
I20260809 04:36:00.532352 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:36:00.532702 19824 external_mini_cluster.cc:1428] Setting key 19eded0737b8a3008f5cb0fd18cfd8d3
I20260809 04:36:00.535036 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:44117
--builtin_ntp_servers=127.19.92.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 04:36:00.535108 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:00.526649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:59497 (local address 127.19.92.62:44117)
0809 04:36:00.526788 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:00.526791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:00.527947 (+ 1156us) server_negotiation.cc:407] Connection header received
0809 04:36:00.529418 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:00.529422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:00.529490 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:00.529568 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:00.530626 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.531182 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.532792 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.533085 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.533236 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:00.534624 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:00.534628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:00.534631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:36:00.534652 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:00.534676 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:36:00.534687 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:00.534765 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:00.534888 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 04:36:00.534960 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:36:00.536089 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:00.527060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:44117 (local address 127.19.92.2:59497)
0809 04:36:00.527780 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:00.527824 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:00.529142 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:00.529776 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:00.529785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:00.529886 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:00.530457 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:00.530474 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.531327 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.531330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:00.532635 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:00.532652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.533323 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.533328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:00.533536 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:00.534300 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:36:00.534302 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:00.534370 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:36:00.534465 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:00.534764 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:00.534770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:00.534774 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:36:00.534870 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 04:36:00.535172 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260809 04:36:00.537395 1586 heartbeater.cc:347] Connected to a master server at 127.19.92.62:44117
I20260809 04:36:00.537679 1586 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:00.538314 1586 heartbeater.cc:511] Master 127.19.92.62:44117 requested a full tablet report, sending...
I20260809 04:36:00.539305 1076 ts_manager.cc:194] Registered new tserver with Master: 5f75b63d52494b3180a7902b4a4555ad (127.19.92.2:36423)
I20260809 04:36:00.539875 1076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.2:59497
W20260809 04:36:00.651839 1591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:00.652105 1591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:00.652148 1591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:00.656165 1591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:00.656219 1591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:00.656279 1591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:36:00.660993 1591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:44117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:00.662060 1591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:00.663426 1591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:36:00.672796 1591 server_base.cc:1034] running on GCE node
W20260809 04:36:00.672783 1601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.672760 1600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.672761 1603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:00.673365 1591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:00.673940 1591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:00.675115 1591 hybrid_clock.cc:648] HybridClock initialized: now 1786250160675100 us; error 29 us; skew 500 ppm
I20260809 04:36:00.676970 1591 webserver.cc:460] Webserver started at http://127.19.92.3:39251/ using document root <none> and password file <none>
I20260809 04:36:00.677619 1591 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:00.677687 1591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:00.677857 1591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:00.680058 1591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "4d4ac30b7af645f7b47cc1a61e61b0ff"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "7c2668b447261fa066461478fba47498"
server_key_iv: "576d0970bbc4722fed7b142fd2da093f"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.680666 1591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "4d4ac30b7af645f7b47cc1a61e61b0ff"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "7c2668b447261fa066461478fba47498"
server_key_iv: "576d0970bbc4722fed7b142fd2da093f"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.684456 1591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:36:00.687067 1613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.688246 1591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:36:00.688349 1591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "4d4ac30b7af645f7b47cc1a61e61b0ff"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "7c2668b447261fa066461478fba47498"
server_key_iv: "576d0970bbc4722fed7b142fd2da093f"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.688468 1591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:00.704205 1591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:00.704948 1591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:00.705123 1591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:00.705794 1591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:00.706740 1591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:00.706790 1591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.706843 1591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:00.706868 1591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.715837 1591 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:46797
I20260809 04:36:00.715863 1797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:46797 every 8 connection(s)
I20260809 04:36:00.716751 1591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:36:00.723146 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 1591
I20260809 04:36:00.723266 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:36:00.723577 19824 external_mini_cluster.cc:1428] Setting key 560c429e6d0c358a4c6c3e52d18e5eb2
I20260809 04:36:00.726225 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:00.718577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:46885 (local address 127.19.92.62:44117)
0809 04:36:00.718703 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:00.718707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:00.719767 (+ 1060us) server_negotiation.cc:407] Connection header received
0809 04:36:00.721129 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:00.721133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:00.721212 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:00.721286 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:00.722183 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.722718 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.724221 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.724550 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.724636 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:00.725835 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:00.725838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:00.725840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:36:00.725852 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:00.725872 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:36:00.725879 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:00.725941 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:00.726065 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 04:36:00.726114 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:36:00.727031 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:00.718975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:44117 (local address 127.19.92.3:46885)
0809 04:36:00.719616 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:00.719653 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:00.720902 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:00.721463 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:00.721469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:00.721552 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:00.722043 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:00.722057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.722833 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.722836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:00.724097 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:00.724108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:00.724695 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:00.724700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:00.724896 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:00.725549 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:36:00.725551 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:00.725603 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:36:00.725685 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:00.725945 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:00.725951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:00.725956 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 04:36:00.726058 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 04:36:00.726280 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260809 04:36:00.728015 1799 heartbeater.cc:347] Connected to a master server at 127.19.92.62:44117
I20260809 04:36:00.728223 1799 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:00.728698 1799 heartbeater.cc:511] Master 127.19.92.62:44117 requested a full tablet report, sending...
I20260809 04:36:00.729723 1076 ts_manager.cc:194] Registered new tserver with Master: 4d4ac30b7af645f7b47cc1a61e61b0ff (127.19.92.3:46797)
I20260809 04:36:00.730279 1076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.3:46885
I20260809 04:36:00.737408 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 04:36:00.740020 1143 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:36:00.739194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.19.92.62:44117)
0809 04:36:00.739338 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:00.739342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:00.739469 (+ 127us) server_negotiation.cc:407] Connection header received
0809 04:36:00.739571 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:00.739573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:00.739614 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:00.739676 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:00.739871 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58792: BlockingRecv error: recv got EOF from 127.0.0.1:58792 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:36:00.740088 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 1172
I20260809 04:36:00.755378 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 1379
I20260809 04:36:00.772102 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 1591
I20260809 04:36:00.782837 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 1024
2026-08-09T04:36:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (902 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T04:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:36:00Z Disabled control of system clock
I20260809 04:36:00.820359 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36849
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 04:36:00.936123 1817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:00.936405 1817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:00.936450 1817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:00.940385 1817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:36:00.940475 1817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:00.940491 1817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:00.940506 1817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:36:00.940526 1817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:36:00.945256 1817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:00.946349 1817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:00.947744 1817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:00.956166 1827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.956156 1828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:00.956178 1832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:00.956207 1817 server_base.cc:1034] running on GCE node
I20260809 04:36:00.956972 1817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:00.958096 1817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:00.959245 1817 hybrid_clock.cc:648] HybridClock initialized: now 1786250160959223 us; error 42 us; skew 500 ppm
I20260809 04:36:00.961061 1817 webserver.cc:460] Webserver started at http://127.19.92.62:39797/ using document root <none> and password file <none>
I20260809 04:36:00.961651 1817 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:00.961715 1817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:00.961997 1817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:00.964020 1817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "a19420f1594a43bf8cd11007d41aa47a"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "917c7b5d9f75e140617b640cd7fbab2b"
server_key_iv: "e36a7a99f96bb6539ab724e32ab48092"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.964577 1817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "a19420f1594a43bf8cd11007d41aa47a"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "917c7b5d9f75e140617b640cd7fbab2b"
server_key_iv: "e36a7a99f96bb6539ab724e32ab48092"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.968183 1817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:36:00.970552 1839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:00.971820 1817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:36:00.971932 1817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "a19420f1594a43bf8cd11007d41aa47a"
format_stamp: "Formatted at 2026-08-09 04:36:00 on dist-test-slave-rmhg"
server_key: "917c7b5d9f75e140617b640cd7fbab2b"
server_key_iv: "e36a7a99f96bb6539ab724e32ab48092"
server_key_version: "encryptionkey@0"
I20260809 04:36:00.972046 1817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:00.976682 1817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:00.977291 1817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:00.977456 1817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:00.984086 1817 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:36849
I20260809 04:36:00.984097 1923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:36849 every 8 connection(s)
I20260809 04:36:00.985064 1817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:36:00.986238 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 1817
I20260809 04:36:00.986354 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:36:00.986692 19824 external_mini_cluster.cc:1428] Setting key bb565177b55fcb6a4b514e26fdd18101
I20260809 04:36:00.988946 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:00.992543 1931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:00.988256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.19.92.62:36849)
0809 04:36:00.989010 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:00.989037 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:00.989115 (+ 78us) server_negotiation.cc:407] Connection header received
0809 04:36:00.989823 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:00.989840 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:00.990218 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:00.990507 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:00.991301 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:00.991305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:00.991316 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:36:00.991326 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:00.991377 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:36:00.991392 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:00.991511 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:00.991587 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 04:36:00.991834 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":149,"threads_started":1}
I20260809 04:36:00.995110 1925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a: Bootstrap starting.
I20260809 04:36:00.997191 1925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:00.997913 1925 log.cc:824] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:00.999737 1925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a: No bootstrap required, opened a new log
I20260809 04:36:01.003134 1925 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19420f1594a43bf8cd11007d41aa47a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36849 } }
I20260809 04:36:01.003345 1925 raft_consensus.cc:374] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:01.003393 1925 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19420f1594a43bf8cd11007d41aa47a, State: Initialized, Role: FOLLOWER
I20260809 04:36:01.003798 1925 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19420f1594a43bf8cd11007d41aa47a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36849 } }
I20260809 04:36:01.003917 1925 raft_consensus.cc:388] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:36:01.003963 1925 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:36:01.004057 1925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:01.004947 1925 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19420f1594a43bf8cd11007d41aa47a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36849 } }
I20260809 04:36:01.005329 1925 leader_election.cc:302] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a19420f1594a43bf8cd11007d41aa47a; no voters:
I20260809 04:36:01.005656 1925 leader_election.cc:288] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:36:01.005779 1935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:36:01.006011 1935 raft_consensus.cc:686] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [term 1 LEADER]: Becoming Leader. State: Replica: a19420f1594a43bf8cd11007d41aa47a, State: Running, Role: LEADER
I20260809 04:36:01.006366 1935 consensus_queue.cc:237] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19420f1594a43bf8cd11007d41aa47a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36849 } }
I20260809 04:36:01.006856 1925 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:36:01.008070 1936 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a19420f1594a43bf8cd11007d41aa47a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19420f1594a43bf8cd11007d41aa47a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36849 } } }
I20260809 04:36:01.008200 1936 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [sys.catalog]: This master's current role is: LEADER
I20260809 04:36:01.008515 1937 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19420f1594a43bf8cd11007d41aa47a. Latest consensus state: current_term: 1 leader_uuid: "a19420f1594a43bf8cd11007d41aa47a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19420f1594a43bf8cd11007d41aa47a" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36849 } } }
I20260809 04:36:01.008586 1944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:36:01.008700 1937 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a [sys.catalog]: This master's current role is: LEADER
I20260809 04:36:01.011669 1944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:36:01.017364 1944 catalog_manager.cc:1347] Generated new cluster ID: dfd512ab6a4e4dcb96e4b9b91b469757
I20260809 04:36:01.017448 1944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:36:01.036351 1944 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:36:01.036674 1944 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:36:01.041661 1944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a19420f1594a43bf8cd11007d41aa47a: Generated new TSK 0
I20260809 04:36:01.042284 1944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:36:01.049259 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36849
--builtin_ntp_servers=127.19.92.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 04:36:01.164022 1958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:01.164274 1958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:01.164314 1958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:01.168382 1958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:01.168444 1958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:01.168511 1958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:36:01.173285 1958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:01.174388 1958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:01.175894 1958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:01.185336 1968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:01.185384 1958 server_base.cc:1034] running on GCE node
W20260809 04:36:01.185323 1970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:01.185326 1966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:01.186012 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:01.186632 1958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:01.187811 1958 hybrid_clock.cc:648] HybridClock initialized: now 1786250161187785 us; error 38 us; skew 500 ppm
I20260809 04:36:01.189680 1958 webserver.cc:460] Webserver started at http://127.19.92.1:35703/ using document root <none> and password file <none>
I20260809 04:36:01.190249 1958 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:01.190318 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:01.190524 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:01.192562 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "f1783d28fe3d4887bc7418b5f12ae0f9"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "a794a97c6aeb8fee528410a61e5d0974"
server_key_iv: "1be5b03df36da2e91a74a48dc314b8da"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.193114 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "f1783d28fe3d4887bc7418b5f12ae0f9"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "a794a97c6aeb8fee528410a61e5d0974"
server_key_iv: "1be5b03df36da2e91a74a48dc314b8da"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.196681 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:36:01.199270 1978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.200489 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:36:01.200604 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "f1783d28fe3d4887bc7418b5f12ae0f9"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "a794a97c6aeb8fee528410a61e5d0974"
server_key_iv: "1be5b03df36da2e91a74a48dc314b8da"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.200709 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:01.205565 1958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:01.206241 1958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:01.206414 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:01.207000 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:01.208045 1958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:01.208093 1958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.208145 1958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:01.208171 1958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.216838 1958 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:41759
I20260809 04:36:01.216859 2138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:41759 every 8 connection(s)
I20260809 04:36:01.217823 1958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:36:01.224013 1931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:01.219706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:57481 (local address 127.19.92.62:36849)
0809 04:36:01.219853 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:01.219856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:01.221001 (+ 1145us) server_negotiation.cc:407] Connection header received
0809 04:36:01.222332 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:01.222336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:01.222384 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:01.222463 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:01.223566 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:01.223569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:01.223571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:36:01.223584 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:01.223604 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:36:01.223612 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:01.223692 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:01.223825 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 04:36:01.223870 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:36:01.224918 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:01.220124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36849 (local address 127.19.92.1:57481)
0809 04:36:01.220854 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:01.220898 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:01.222115 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:01.222647 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:01.222654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:01.222747 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:01.223278 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:36:01.223279 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:01.223322 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:36:01.223393 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:01.223684 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:01.223694 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:01.223701 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:36:01.223866 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 04:36:01.224136 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260809 04:36:01.226226 2139 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36849
I20260809 04:36:01.226432 2139 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:01.226750 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 1958
I20260809 04:36:01.226861 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:36:01.226838 2139 heartbeater.cc:511] Master 127.19.92.62:36849 requested a full tablet report, sending...
I20260809 04:36:01.227157 19824 external_mini_cluster.cc:1428] Setting key 8dbe835640c1a5c478ae3a8c3477235e
I20260809 04:36:01.228449 1864 ts_manager.cc:194] Registered new tserver with Master: f1783d28fe3d4887bc7418b5f12ae0f9 (127.19.92.1:41759)
I20260809 04:36:01.229280 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36849
--builtin_ntp_servers=127.19.92.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 04:36:01.341792 2143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:01.342061 2143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:01.342120 2143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:01.346165 2143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:01.346221 2143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:01.346298 2143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:36:01.351020 2143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:01.352105 2143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:01.353524 2143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:01.362896 2157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:01.362985 2143 server_base.cc:1034] running on GCE node
W20260809 04:36:01.362929 2150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:01.362929 2155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:01.363602 2143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:01.364189 2143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:01.365348 2143 hybrid_clock.cc:648] HybridClock initialized: now 1786250161365318 us; error 41 us; skew 500 ppm
I20260809 04:36:01.367219 2143 webserver.cc:460] Webserver started at http://127.19.92.2:36523/ using document root <none> and password file <none>
I20260809 04:36:01.367801 2143 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:01.367871 2143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:01.368067 2143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:01.370253 2143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "19e718f303b44409857829b5c5e63172"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "07bc9d2c2e740d256c2d98585659cfd9"
server_key_iv: "1df9262e6e9cc428720efbdcece1ea8b"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.370862 2143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "19e718f303b44409857829b5c5e63172"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "07bc9d2c2e740d256c2d98585659cfd9"
server_key_iv: "1df9262e6e9cc428720efbdcece1ea8b"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.374888 2143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 04:36:01.377723 2167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.379110 2143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:36:01.379235 2143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "19e718f303b44409857829b5c5e63172"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "07bc9d2c2e740d256c2d98585659cfd9"
server_key_iv: "1df9262e6e9cc428720efbdcece1ea8b"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.379347 2143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:01.384133 2143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:01.384779 2143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:01.384948 2143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:01.385548 2143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:01.386482 2143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:01.386529 2143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.386581 2143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:01.386606 2143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.395128 2143 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:42463
I20260809 04:36:01.395149 2352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:42463 every 8 connection(s)
I20260809 04:36:01.396080 2143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:36:01.402099 1931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:01.397924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:59409 (local address 127.19.92.62:36849)
0809 04:36:01.398105 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:01.398109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:01.399233 (+ 1124us) server_negotiation.cc:407] Connection header received
0809 04:36:01.400530 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:01.400533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:01.400576 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:01.400655 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:01.401734 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:01.401737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:01.401740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:36:01.401751 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:01.401772 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:36:01.401780 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:01.401857 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:01.401948 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 04:36:01.401990 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 04:36:01.402948 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:01.398411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36849 (local address 127.19.92.2:59409)
0809 04:36:01.399082 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:01.399123 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:01.400319 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:01.400802 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:01.400809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:01.400900 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:01.401479 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:36:01.401480 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:01.401523 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:36:01.401596 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:01.401848 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:01.401853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:01.401857 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:36:01.401981 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 04:36:01.402188 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260809 04:36:01.404044 2353 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36849
I20260809 04:36:01.404248 2353 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:01.404628 2353 heartbeater.cc:511] Master 127.19.92.62:36849 requested a full tablet report, sending...
I20260809 04:36:01.405671 1864 ts_manager.cc:194] Registered new tserver with Master: 19e718f303b44409857829b5c5e63172 (127.19.92.2:42463)
I20260809 04:36:01.406371 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 2143
I20260809 04:36:01.406488 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:36:01.406780 19824 external_mini_cluster.cc:1428] Setting key 2d96b706045e270f4607b2727c73e5f3
I20260809 04:36:01.408597 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36849
--builtin_ntp_servers=127.19.92.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 04:36:01.523665 2357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:01.523921 2357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:01.523962 2357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:01.527947 2357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:01.528004 2357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:01.528064 2357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:36:01.532795 2357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:36849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:01.534010 2357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:01.535408 2357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:01.544773 2368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:01.544775 2365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:01.544764 2366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:01.544978 2357 server_base.cc:1034] running on GCE node
I20260809 04:36:01.545493 2357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:01.546080 2357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:01.547236 2357 hybrid_clock.cc:648] HybridClock initialized: now 1786250161547218 us; error 31 us; skew 500 ppm
I20260809 04:36:01.549031 2357 webserver.cc:460] Webserver started at http://127.19.92.3:44171/ using document root <none> and password file <none>
I20260809 04:36:01.549606 2357 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:01.549677 2357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:01.549878 2357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:01.551856 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "0acb8350d6954e0ca34a1db78d920295"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "c03ba2d425f39efbd57ed82d060c62a8"
server_key_iv: "65f6314468950ee94e67168a21545cdc"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.552392 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "0acb8350d6954e0ca34a1db78d920295"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "c03ba2d425f39efbd57ed82d060c62a8"
server_key_iv: "65f6314468950ee94e67168a21545cdc"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.555934 2357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 04:36:01.558350 2375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.559617 2357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:36:01.559729 2357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "0acb8350d6954e0ca34a1db78d920295"
format_stamp: "Formatted at 2026-08-09 04:36:01 on dist-test-slave-rmhg"
server_key: "c03ba2d425f39efbd57ed82d060c62a8"
server_key_iv: "65f6314468950ee94e67168a21545cdc"
server_key_version: "encryptionkey@0"
I20260809 04:36:01.559818 2357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:01.564265 2357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:01.564844 2357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:01.565003 2357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:01.565548 2357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:01.566430 2357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:01.566478 2357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.566526 2357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:01.566555 2357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:01.575201 2357 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:33861
I20260809 04:36:01.575235 2551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:33861 every 8 connection(s)
I20260809 04:36:01.576176 2357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:36:01.582526 1931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:01.578147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:56113 (local address 127.19.92.62:36849)
0809 04:36:01.578322 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:01.578327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:01.579483 (+ 1156us) server_negotiation.cc:407] Connection header received
0809 04:36:01.580760 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:01.580763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:01.580802 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:01.580876 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:01.582013 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:01.582016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:01.582018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:36:01.582028 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:01.582045 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:36:01.582053 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:01.582124 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:01.582302 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 04:36:01.582374 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:36:01.583365 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:01.578601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36849 (local address 127.19.92.3:56113)
0809 04:36:01.579331 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:01.579374 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:01.580578 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:01.581059 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:01.581066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:01.581181 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:01.581758 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:36:01.581760 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:01.581803 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:36:01.581877 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:01.582123 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:01.582131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:01.582137 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:36:01.582296 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 04:36:01.582553 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":110,"threads_started":1}
I20260809 04:36:01.584549 2553 heartbeater.cc:347] Connected to a master server at 127.19.92.62:36849
I20260809 04:36:01.584774 2553 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:01.585227 2553 heartbeater.cc:511] Master 127.19.92.62:36849 requested a full tablet report, sending...
I20260809 04:36:01.585768 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 2357
I20260809 04:36:01.585862 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:36:01.586148 19824 external_mini_cluster.cc:1428] Setting key ea1188fe0fd9b4d1ff54f2072c264882
I20260809 04:36:01.586339 1864 ts_manager.cc:194] Registered new tserver with Master: 0acb8350d6954e0ca34a1db78d920295 (127.19.92.3:33861)
I20260809 04:36:01.588157 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 04:36:01.590953 1931 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:36:01.590139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.19.92.62:36849)
0809 04:36:01.590261 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:01.590265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:01.590449 (+ 184us) server_negotiation.cc:407] Connection header received
0809 04:36:01.590555 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:01.590557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:01.590589 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:01.590650 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:01.590837 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35750: BlockingRecv error: recv got EOF from 127.0.0.1:35750 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 04:36:01.591264 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 1958
I20260809 04:36:01.603968 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 2143
I20260809 04:36:01.617496 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 2357
I20260809 04:36:01.634929 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 1817
2026-08-09T04:36:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (851 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:36:01 dist-test-slave-rmhg krb5kdc[2567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:36:01 dist-test-slave-rmhg krb5kdc[2567](info): set up 2 sockets
Aug 09 04:36:01 dist-test-slave-rmhg krb5kdc[2567](info): commencing operation
krb5kdc: starting...
W20260809 04:36:02.760399 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:36:02 dist-test-slave-rmhg krb5kdc[2567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:36:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:36:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:36:02.886489 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:43371
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:43371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:36:02.999845 2584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:03.000111 2584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:03.000168 2584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:03.004169 2584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:36:03.004231 2584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:03.004258 2584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:03.004279 2584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:36:03.004299 2584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:36:03.009678 2584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:43371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:03.010830 2584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:03.011963 2584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:36:03.020804 2584 server_base.cc:1034] running on GCE node
W20260809 04:36:03.020788 2592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:03.020974 2597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:03.020793 2593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:03.021620 2584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:03.022722 2584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:03.023877 2584 hybrid_clock.cc:648] HybridClock initialized: now 1786250163023855 us; error 41 us; skew 500 ppm
Aug 09 04:36:03 dist-test-slave-rmhg krb5kdc[2567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:36:03.032264 2584 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:36:03.033263 2584 webserver.cc:460] Webserver started at http://127.19.92.62:44629/ using document root <none> and password file <none>
I20260809 04:36:03.033818 2584 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:03.033881 2584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:03.034101 2584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:03.036164 2584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "1c0b5d3d0be44f66a84d7df28957d838"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "4b0c1093a5316e241df9c1e6accfe25d"
server_key_iv: "91c6c002c9db7ad1c54b83d89f9afd47"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.036698 2584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "1c0b5d3d0be44f66a84d7df28957d838"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "4b0c1093a5316e241df9c1e6accfe25d"
server_key_iv: "91c6c002c9db7ad1c54b83d89f9afd47"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.040309 2584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:36:03.042627 2608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.043809 2584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:36:03.043921 2584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "1c0b5d3d0be44f66a84d7df28957d838"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "4b0c1093a5316e241df9c1e6accfe25d"
server_key_iv: "91c6c002c9db7ad1c54b83d89f9afd47"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.044023 2584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:03.062348 2584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:03.065728 2584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:03.065937 2584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:03.072844 2584 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:43371
I20260809 04:36:03.072922 2695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:43371 every 8 connection(s)
I20260809 04:36:03.073922 2584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:36:03.076726 2696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:03.082185 2696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838: Bootstrap starting.
I20260809 04:36:03.083950 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 2584
I20260809 04:36:03.084075 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:36:03.084324 19824 external_mini_cluster.cc:1428] Setting key 61263ab98f1b440e37d3ebcc86e5c877
I20260809 04:36:03.084676 2696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:03.085565 2696 log.cc:824] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:03.087642 2696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838: No bootstrap required, opened a new log
I20260809 04:36:03.091732 2696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0b5d3d0be44f66a84d7df28957d838" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43371 } }
I20260809 04:36:03.092062 2696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:03.092125 2696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0b5d3d0be44f66a84d7df28957d838, State: Initialized, Role: FOLLOWER
I20260809 04:36:03.092748 2696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0b5d3d0be44f66a84d7df28957d838" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43371 } }
I20260809 04:36:03.092939 2696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:36:03.093024 2696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:36:03.093139 2696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 04:36:03 dist-test-slave-rmhg krb5kdc[2567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:36:03.094218 2696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0b5d3d0be44f66a84d7df28957d838" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43371 } }
I20260809 04:36:03.094586 2696 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c0b5d3d0be44f66a84d7df28957d838; no voters:
I20260809 04:36:03.094964 2696 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:36:03.095083 2705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:36:03.095564 2705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0b5d3d0be44f66a84d7df28957d838, State: Running, Role: LEADER
I20260809 04:36:03.095881 2705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0b5d3d0be44f66a84d7df28957d838" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43371 } }
I20260809 04:36:03.096401 2696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:36:03.097131 2705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0b5d3d0be44f66a84d7df28957d838. Latest consensus state: current_term: 1 leader_uuid: "1c0b5d3d0be44f66a84d7df28957d838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0b5d3d0be44f66a84d7df28957d838" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43371 } } }
I20260809 04:36:03.097786 2705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [sys.catalog]: This master's current role is: LEADER
I20260809 04:36:03.097662 2707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c0b5d3d0be44f66a84d7df28957d838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0b5d3d0be44f66a84d7df28957d838" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43371 } } }
I20260809 04:36:03.098064 2707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838 [sys.catalog]: This master's current role is: LEADER
I20260809 04:36:03.098351 2719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:36:03.099828 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.085472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.19.92.62:43371)
0809 04:36:03.085889 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.085905 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.085940 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:36:03.086564 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.086581 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.086898 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.087190 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:03.088100 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.089202 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.090439 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.091108 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.091240 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.094202 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:03.094208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:03.094223 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:03.094256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:03.096173 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.096608 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.096614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.096619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.096753 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.097036 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.097040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.097042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.097610 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:03.098416 (+ 806us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.098585 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.098890 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260809 04:36:03.101527 2719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:36:03.107007 2719 catalog_manager.cc:1347] Generated new cluster ID: eb45ff79dd854022b192d0f406e5bc7e
I20260809 04:36:03.107127 2719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:36:03.113211 2719 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:36:03.114219 2719 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:36:03.123103 2719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838: Generated new TSK 0
I20260809 04:36:03.123776 2719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:36:03.157799 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43371
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:36:03.272123 2748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:03.272418 2748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:03.272461 2748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:03.276527 2748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:03.276592 2748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:03.276652 2748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:36:03.281982 2748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:03.283071 2748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:03.284159 2748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:03.294150 2758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:03.294168 2760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:03.294150 2757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:03.294405 2748 server_base.cc:1034] running on GCE node
I20260809 04:36:03.294894 2748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:03.295567 2748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:03.296757 2748 hybrid_clock.cc:648] HybridClock initialized: now 1786250163296721 us; error 45 us; skew 500 ppm
Aug 09 04:36:03 dist-test-slave-rmhg krb5kdc[2567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:36:03.299707 2748 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:36:03.300631 2748 webserver.cc:460] Webserver started at http://127.19.92.1:33021/ using document root <none> and password file <none>
I20260809 04:36:03.301234 2748 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:03.301309 2748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:03.301529 2748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:03.303598 2748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "37de81dd1e5840d1986d534597d96436"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "d71fd75853bbbb752845f35dd339dbc9"
server_key_iv: "1c02179801dcf9329a4d5ab6fd73c34e"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.304147 2748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "37de81dd1e5840d1986d534597d96436"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "d71fd75853bbbb752845f35dd339dbc9"
server_key_iv: "1c02179801dcf9329a4d5ab6fd73c34e"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.307926 2748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:36:03.310510 2770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.311775 2748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:36:03.311885 2748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "37de81dd1e5840d1986d534597d96436"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "d71fd75853bbbb752845f35dd339dbc9"
server_key_iv: "1c02179801dcf9329a4d5ab6fd73c34e"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.312001 2748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:03.335497 2748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:03.338824 2748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:03.339011 2748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:03.339628 2748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:03.340467 2748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:03.340514 2748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.340559 2748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:03.340587 2748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.349470 2748 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:35615
I20260809 04:36:03.349507 2948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:35615 every 8 connection(s)
I20260809 04:36:03.350378 2748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:36:03.355223 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 2748
I20260809 04:36:03.355330 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:36:03.355629 19824 external_mini_cluster.cc:1428] Setting key fd35fd727991915f026fd977f913f1e3
Aug 09 04:36:03 dist-test-slave-rmhg krb5kdc[2567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:36:03.364480 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.352147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:50639 (local address 127.19.92.62:43371)
0809 04:36:03.352278 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.352283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.353178 (+ 895us) server_negotiation.cc:407] Connection header received
0809 04:36:03.354524 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.354527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.354597 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.354672 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:03.355899 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.356551 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.357900 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.358229 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.358336 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.360733 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:03.360737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:03.360740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:03.360769 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:03.362346 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.362857 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.362861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.362863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.362964 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.363288 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.363292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.363293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.363556 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:03.363667 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.364139 (+ 472us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.364278 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:36:03.365296 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.352518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43371 (local address 127.19.92.1:50639)
0809 04:36:03.353005 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:03.353049 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:03.354337 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:03.354829 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:03.354836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:03.355199 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:03.355753 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.355770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.356694 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.356697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.357774 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.357785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.358361 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.358364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.358544 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.359144 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:36:03.359162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:03.360560 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:03.362482 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:03.362488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:03.362502 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:03.362760 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:03.363058 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:03.363060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:03.363061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:03.363215 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:03.363649 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:03.363654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:03.363905 (+ 251us) client_negotiation.cc:769] Sending connection context
0809 04:36:03.364072 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 04:36:03.364341 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:36:03.366539 2949 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43371
I20260809 04:36:03.366798 2949 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:03.367368 2949 heartbeater.cc:511] Master 127.19.92.62:43371 requested a full tablet report, sending...
I20260809 04:36:03.368888 2638 ts_manager.cc:194] Registered new tserver with Master: 37de81dd1e5840d1986d534597d96436 (127.19.92.1:35615)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:36:03.370324 2638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:50639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:36:03.386370 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43371
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:36:03.500526 2955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:03.500792 2955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:03.500855 2955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:03.504782 2955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:03.504839 2955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:03.504920 2955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:36:03.510067 2955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:03.511086 2955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:03.512086 2955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:03.521389 2965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:03.521510 2967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:03.521389 2964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:03.521656 2955 server_base.cc:1034] running on GCE node
I20260809 04:36:03.522037 2955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:03.522598 2955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:03.523748 2955 hybrid_clock.cc:648] HybridClock initialized: now 1786250163523726 us; error 35 us; skew 500 ppm
Aug 09 04:36:03 dist-test-slave-rmhg krb5kdc[2567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:36:03.526412 2955 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:36:03.527289 2955 webserver.cc:460] Webserver started at http://127.19.92.2:34271/ using document root <none> and password file <none>
I20260809 04:36:03.527814 2955 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:03.527869 2955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:03.528036 2955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:03.530109 2955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "4587816bdca146e3b74f2185f5996828"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "38f438ce8e4a9466e641373626849a42"
server_key_iv: "ec8ba3249a1462250f3658df2b203bda"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.530653 2955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "4587816bdca146e3b74f2185f5996828"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "38f438ce8e4a9466e641373626849a42"
server_key_iv: "ec8ba3249a1462250f3658df2b203bda"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.534587 2955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:36:03.537092 2979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.538374 2955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:36:03.538511 2955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "4587816bdca146e3b74f2185f5996828"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "38f438ce8e4a9466e641373626849a42"
server_key_iv: "ec8ba3249a1462250f3658df2b203bda"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.538627 2955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:03.555454 2955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:03.558658 2955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:03.558838 2955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:03.559437 2955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:03.560305 2955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:03.560353 2955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.560408 2955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:03.560443 2955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.569101 2955 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:41613
I20260809 04:36:03.569123 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:41613 every 8 connection(s)
I20260809 04:36:03.570120 2955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:36:03.573855 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 2955
I20260809 04:36:03.574015 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:36:03.574361 19824 external_mini_cluster.cc:1428] Setting key 12de12e4a460be4ccc6b1d1c0caeb068
Aug 09 04:36:03 dist-test-slave-rmhg krb5kdc[2567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:36:03.585261 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.572023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:33907 (local address 127.19.92.62:43371)
0809 04:36:03.572204 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.572209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.573196 (+ 987us) server_negotiation.cc:407] Connection header received
0809 04:36:03.575051 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.575055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.575119 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.575199 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:03.576543 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.577285 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.578783 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.579018 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.579096 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.581618 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:03.581622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:03.581625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:03.581659 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:03.583162 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.583658 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.583662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.583664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.583761 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.584074 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.584077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.584079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.584319 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:03.584454 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.584916 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.585048 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:36:03.586199 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.572431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43371 (local address 127.19.92.2:33907)
0809 04:36:03.573010 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:03.573061 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:03.574803 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:03.575387 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:03.575395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:03.575862 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:03.576393 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.576407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.577408 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.577410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.578646 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.578658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.579169 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.579173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.579374 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.579952 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:36:03.579969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:03.581437 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:03.583281 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:03.583286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:03.583301 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:03.583560 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:03.583842 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:03.583843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:03.583845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:03.583980 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:03.584425 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:03.584430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:03.584680 (+ 250us) client_negotiation.cc:769] Sending connection context
0809 04:36:03.584837 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 04:36:03.585118 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":133,"threads_started":1}
I20260809 04:36:03.587430 3154 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:36:03.587693 3154 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:03.588266 3154 heartbeater.cc:511] Master 127.19.92.62:43371 requested a full tablet report, sending...
I20260809 04:36:03.589570 2638 ts_manager.cc:194] Registered new tserver with Master: 4587816bdca146e3b74f2185f5996828 (127.19.92.2:41613)
I20260809 04:36:03.590179 2638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:33907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:36:03.602095 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43371
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:36:03.715543 3162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:03.715837 3162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:03.715898 3162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:03.719944 3162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:03.720005 3162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:03.720088 3162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:36:03.725718 3162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:03.726841 3162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:03.727941 3162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:03.737648 3171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:03.737654 3168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:03.737653 3169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:03.737772 3162 server_base.cc:1034] running on GCE node
I20260809 04:36:03.738322 3162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:03.738956 3162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:03.740124 3162 hybrid_clock.cc:648] HybridClock initialized: now 1786250163740102 us; error 35 us; skew 500 ppm
Aug 09 04:36:03 dist-test-slave-rmhg krb5kdc[2567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:36:03.742923 3162 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:36:03.743808 3162 webserver.cc:460] Webserver started at http://127.19.92.3:35407/ using document root <none> and password file <none>
I20260809 04:36:03.744357 3162 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:03.744438 3162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:03.744650 3162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:36:03.746721 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "9320af207b364f2793f0729f989bed87"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "19c51363839ca9ec1b0bc38f60f00251"
server_key_iv: "096af3822599c241a256fbee52d82835"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.747309 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "9320af207b364f2793f0729f989bed87"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "19c51363839ca9ec1b0bc38f60f00251"
server_key_iv: "096af3822599c241a256fbee52d82835"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.751036 3162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 04:36:03.753530 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.754720 3162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:36:03.754832 3162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "9320af207b364f2793f0729f989bed87"
format_stamp: "Formatted at 2026-08-09 04:36:03 on dist-test-slave-rmhg"
server_key: "19c51363839ca9ec1b0bc38f60f00251"
server_key_iv: "096af3822599c241a256fbee52d82835"
server_key_version: "encryptionkey@0"
I20260809 04:36:03.754932 3162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:03.775728 3162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:03.779021 3162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:03.779301 3162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:03.780117 3162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:03.780992 3162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:03.781041 3162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.781081 3162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:03.781095 3162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:03.790341 3162 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:35573
I20260809 04:36:03.790369 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:35573 every 8 connection(s)
I20260809 04:36:03.791237 3162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:36:03.799606 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 3162
I20260809 04:36:03.799737 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:36:03.800011 19824 external_mini_cluster.cc:1428] Setting key 33ef3949a9b683c63121e9a54ada287b
Aug 09 04:36:03 dist-test-slave-rmhg krb5kdc[2567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:36:03.804392 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.792994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:53753 (local address 127.19.92.62:43371)
0809 04:36:03.793126 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.793129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.793995 (+ 866us) server_negotiation.cc:407] Connection header received
0809 04:36:03.795379 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.795383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.795442 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.795515 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:03.796714 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.797342 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.798561 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.798786 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.798873 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.801471 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:03.801474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:03.801476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:03.801499 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:03.802770 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.803219 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.803221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.803222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.803293 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.803536 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.803537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.803538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.803706 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:03.803775 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.804186 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.804275 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:36:03.805254 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.793402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43371 (local address 127.19.92.3:53753)
0809 04:36:03.793849 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:03.793891 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:03.795176 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:03.795701 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:03.795708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:03.796062 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:03.796540 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.796564 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.797457 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.797460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.798450 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.798461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.798920 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.798924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.799087 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.799692 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:36:03.799709 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:03.801343 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:03.802869 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:03.802874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:03.802886 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:03.803139 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:03.803350 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:03.803352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:03.803353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:03.803475 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:03.803776 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:03.803780 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:03.804004 (+ 224us) client_negotiation.cc:769] Sending connection context
0809 04:36:03.804155 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 04:36:03.804398 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260809 04:36:03.806360 3361 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43371
I20260809 04:36:03.806592 3361 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:03.807068 3361 heartbeater.cc:511] Master 127.19.92.62:43371 requested a full tablet report, sending...
I20260809 04:36:03.808063 2638 ts_manager.cc:194] Registered new tserver with Master: 9320af207b364f2793f0729f989bed87 (127.19.92.3:35573)
I20260809 04:36:03.808593 2638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:53753
I20260809 04:36:03.813692 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:36:03.822780 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.815615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.19.92.62:43371)
0809 04:36:03.815752 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.815757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.815895 (+ 138us) server_negotiation.cc:407] Connection header received
0809 04:36:03.816032 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.816035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.816081 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.816144 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:03.816896 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.817485 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.818469 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.818681 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.818766 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.819998 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:03.820002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:03.820005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:03.820032 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:03.821190 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.821588 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.821591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.821593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.821676 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:03.821915 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:03.821919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:03.821920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:03.822153 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:03.822272 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.822515 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.822627 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:36:03.825590 2638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:36:03.828554 2638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:36:03.842821 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.837889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.19.92.1:35615)
0809 04:36:03.838374 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.838388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.838408 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:36:03.838469 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.838474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.838617 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.838774 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:03.839187 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.840150 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.841518 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.842119 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.842275 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.842429 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.842585 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.842678 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":105,"threads_started":1}
I20260809 04:36:03.842813 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.837495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:35615 (local address 127.0.0.1:47234)
0809 04:36:03.838272 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:03.838284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:03.838359 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:03.838956 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:03.838959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:03.838980 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:03.839066 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.839072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.840285 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.840289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.841413 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.841421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.842282 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.842286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.842392 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.842430 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 04:36:03.842538 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 04:36:03.842664 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":96,"threads_started":1}
I20260809 04:36:03.843065 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.837763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.19.92.3:35573)
0809 04:36:03.838296 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.838309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.838334 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:36:03.838411 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.838416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.838618 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.838774 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:03.838999 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.839836 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.841237 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.842121 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.842230 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.842383 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.842812 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.842912 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":159,"threads_started":1}
I20260809 04:36:03.843111 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.836857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.19.92.2:41613)
0809 04:36:03.837297 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.837323 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.837349 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:36:03.837621 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.837627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.837844 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.838019 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:03.838523 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.839559 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.841381 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.842164 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.842248 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.842383 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.842878 (+ 495us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.842970 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":107,"threads_started":1}
I20260809 04:36:03.843269 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.836722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:41613 (local address 127.0.0.1:39290)
0809 04:36:03.837222 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:03.837271 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:03.837394 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:03.838236 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:03.838242 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:03.838266 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:03.838401 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.838408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.839699 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.839702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.841262 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.841274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.842449 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.842453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.842612 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.842652 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 04:36:03.842716 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 04:36:03.842808 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":104,"thread_start_us":87,"threads_started":1}
I20260809 04:36:03.843421 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.836817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:35573 (local address 127.0.0.1:45748)
0809 04:36:03.837444 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:03.837457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:03.837545 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:03.838759 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:03.838762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:03.838782 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:03.838882 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.838888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.839973 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.839977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.841080 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.841089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.842449 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.842453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.842593 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.842648 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 04:36:03.842716 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 04:36:03.843291 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":498,"thread_start_us":116,"threads_started":1}
I20260809 04:36:03.846303 3048 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c (DEFAULT_TABLE table=test-table [id=47a0846d38b04951a43a0fd13d0a5c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:36:03.846668 2841 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c (DEFAULT_TABLE table=test-table [id=47a0846d38b04951a43a0fd13d0a5c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:36:03.846791 3250 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c (DEFAULT_TABLE table=test-table [id=47a0846d38b04951a43a0fd13d0a5c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:36:03.847318 3048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:03.847744 3250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:03.847738 2841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:03.853869 3382 tablet_bootstrap.cc:492] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: Bootstrap starting.
I20260809 04:36:03.855504 3384 tablet_bootstrap.cc:492] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436: Bootstrap starting.
I20260809 04:36:03.855907 3385 tablet_bootstrap.cc:492] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: Bootstrap starting.
I20260809 04:36:03.856596 3382 tablet_bootstrap.cc:654] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:03.857402 3382 log.cc:824] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:03.858053 3385 tablet_bootstrap.cc:654] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:03.859225 3384 tablet_bootstrap.cc:654] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:03.859251 3385 log.cc:824] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:03.859294 3382 tablet_bootstrap.cc:492] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: No bootstrap required, opened a new log
I20260809 04:36:03.859472 3382 ts_tablet_manager.cc:1397] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:36:03.860312 3384 log.cc:824] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:03.861289 3385 tablet_bootstrap.cc:492] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: No bootstrap required, opened a new log
I20260809 04:36:03.861471 3385 ts_tablet_manager.cc:1397] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:36:03.862293 3384 tablet_bootstrap.cc:492] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436: No bootstrap required, opened a new log
I20260809 04:36:03.862743 3384 ts_tablet_manager.cc:1397] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260809 04:36:03.863122 3382 raft_consensus.cc:348] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.863343 3382 raft_consensus.cc:374] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:03.863394 3382 raft_consensus.cc:729] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4587816bdca146e3b74f2185f5996828, State: Initialized, Role: FOLLOWER
I20260809 04:36:03.863824 3382 consensus_queue.cc:260] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.864498 3154 heartbeater.cc:503] Master 127.19.92.62:43371 was elected leader, sending a full tablet report...
I20260809 04:36:03.865350 3385 raft_consensus.cc:348] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.865612 3385 raft_consensus.cc:374] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:03.865660 3385 raft_consensus.cc:729] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9320af207b364f2793f0729f989bed87, State: Initialized, Role: FOLLOWER
I20260809 04:36:03.865824 3382 ts_tablet_manager.cc:1428] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:36:03.866129 3385 consensus_queue.cc:260] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.866500 3384 raft_consensus.cc:348] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.866745 3384 raft_consensus.cc:374] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:03.866816 3361 heartbeater.cc:503] Master 127.19.92.62:43371 was elected leader, sending a full tablet report...
I20260809 04:36:03.866816 3384 raft_consensus.cc:729] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37de81dd1e5840d1986d534597d96436, State: Initialized, Role: FOLLOWER
I20260809 04:36:03.867236 3384 consensus_queue.cc:260] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.867802 2949 heartbeater.cc:503] Master 127.19.92.62:43371 was elected leader, sending a full tablet report...
I20260809 04:36:03.868247 3385 ts_tablet_manager.cc:1428] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260809 04:36:03.869127 3384 ts_tablet_manager.cc:1428] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260809 04:36:03.940780 3396 raft_consensus.cc:482] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:36:03.941025 3396 raft_consensus.cc:504] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.942253 3396 leader_election.cc:288] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4587816bdca146e3b74f2185f5996828 (127.19.92.2:41613), 9320af207b364f2793f0729f989bed87 (127.19.92.3:35573)
I20260809 04:36:03.947176 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.942632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:52857 (local address 127.19.92.2:41613)
0809 04:36:03.942762 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.942767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.942961 (+ 194us) server_negotiation.cc:407] Connection header received
0809 04:36:03.943083 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.943085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.943133 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.943214 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:03.943514 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.944275 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.945858 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.946494 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.946602 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.946671 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.946965 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.947048 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:36:03.947175 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.942663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:52049 (local address 127.19.92.3:35573)
0809 04:36:03.942794 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.942798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.943227 (+ 429us) server_negotiation.cc:407] Connection header received
0809 04:36:03.943393 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.943396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.943439 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.943511 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:03.943726 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.944470 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.945858 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.946494 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.946602 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.946671 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.946965 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.947049 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 04:36:03.947190 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.942507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:35573 (local address 127.19.92.1:52049)
0809 04:36:03.943151 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:03.943161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:03.943283 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:03.943536 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:03.943539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:03.943558 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:03.943638 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.943642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.944562 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.944565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.945720 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.945731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.946618 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.946622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.946806 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.946841 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 04:36:03.946906 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 04:36:03.947019 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":380,"thread_start_us":107,"threads_started":1}
I20260809 04:36:03.947610 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.942505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:41613 (local address 127.19.92.1:52857)
0809 04:36:03.942864 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:03.942879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:03.942989 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:03.943236 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:03.943239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:03.943263 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:03.943413 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.943419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.944379 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.944381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.945720 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:03.945731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.946618 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.946622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:03.946774 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.946829 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 04:36:03.946906 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:36:03.947037 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":92,"thread_start_us":143,"threads_started":1}
I20260809 04:36:03.947960 3288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc2a3a33f1b4007b6f843f75a1b6e5c" candidate_uuid: "37de81dd1e5840d1986d534597d96436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9320af207b364f2793f0729f989bed87" is_pre_election: true
I20260809 04:36:03.948269 3085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc2a3a33f1b4007b6f843f75a1b6e5c" candidate_uuid: "37de81dd1e5840d1986d534597d96436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4587816bdca146e3b74f2185f5996828" is_pre_election: true
I20260809 04:36:03.948302 3288 raft_consensus.cc:2454] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37de81dd1e5840d1986d534597d96436 in term 0.
I20260809 04:36:03.948587 3085 raft_consensus.cc:2454] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37de81dd1e5840d1986d534597d96436 in term 0.
I20260809 04:36:03.948779 2777 leader_election.cc:302] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37de81dd1e5840d1986d534597d96436, 9320af207b364f2793f0729f989bed87; no voters:
I20260809 04:36:03.949046 3396 raft_consensus.cc:2781] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:36:03.949137 3396 raft_consensus.cc:482] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:36:03.949210 3396 raft_consensus.cc:3037] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:03.950241 3396 raft_consensus.cc:504] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.950567 3396 leader_election.cc:288] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [CANDIDATE]: Term 1 election: Requested vote from peers 4587816bdca146e3b74f2185f5996828 (127.19.92.2:41613), 9320af207b364f2793f0729f989bed87 (127.19.92.3:35573)
I20260809 04:36:03.950902 3085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc2a3a33f1b4007b6f843f75a1b6e5c" candidate_uuid: "37de81dd1e5840d1986d534597d96436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4587816bdca146e3b74f2185f5996828"
I20260809 04:36:03.951045 3085 raft_consensus.cc:3037] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:03.952064 3085 raft_consensus.cc:2454] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37de81dd1e5840d1986d534597d96436 in term 1.
I20260809 04:36:03.952282 3288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc2a3a33f1b4007b6f843f75a1b6e5c" candidate_uuid: "37de81dd1e5840d1986d534597d96436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9320af207b364f2793f0729f989bed87"
I20260809 04:36:03.952353 2775 leader_election.cc:302] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37de81dd1e5840d1986d534597d96436, 4587816bdca146e3b74f2185f5996828; no voters:
I20260809 04:36:03.952390 3288 raft_consensus.cc:3037] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:03.952584 3396 raft_consensus.cc:2781] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:36:03.952816 3396 raft_consensus.cc:686] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 1 LEADER]: Becoming Leader. State: Replica: 37de81dd1e5840d1986d534597d96436, State: Running, Role: LEADER
I20260809 04:36:03.953203 3396 consensus_queue.cc:237] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } }
I20260809 04:36:03.953724 3288 raft_consensus.cc:2454] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37de81dd1e5840d1986d534597d96436 in term 1.
I20260809 04:36:03.956682 2637 catalog_manager.cc:5495] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37de81dd1e5840d1986d534597d96436 (127.19.92.1). New cstate: current_term: 1 leader_uuid: "37de81dd1e5840d1986d534597d96436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37de81dd1e5840d1986d534597d96436" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 35615 } health_report { overall_health: HEALTHY } } }
I20260809 04:36:03.968959 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:03.965751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.19.92.1:35615)
0809 04:36:03.965870 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:03.965874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:03.965887 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:36:03.965991 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:03.965993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:03.966048 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:03.966134 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:36:03.966314 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.966808 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.967653 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:03.967864 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:03.967943 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:03.968203 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:36:03.968623 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:36:03.968709 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:03.968750 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 04:36:03.968803 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:36:03.975049 3085 raft_consensus.cc:1261] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 1 FOLLOWER]: Refusing update from remote peer 37de81dd1e5840d1986d534597d96436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:36:03.975047 3288 raft_consensus.cc:1261] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 1 FOLLOWER]: Refusing update from remote peer 37de81dd1e5840d1986d534597d96436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:36:03.975752 3399 consensus_queue.cc:1035] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4587816bdca146e3b74f2185f5996828" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:36:03.975982 3396 consensus_queue.cc:1035] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9320af207b364f2793f0729f989bed87" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:36:03.993140 2637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:36:03.993343 2637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:36:03.995716 2637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c0b5d3d0be44f66a84d7df28957d838: Sending DeleteTablet for 3 replicas of tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c
I20260809 04:36:03.996652 2841 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:36:03 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:47234
I20260809 04:36:03.996652 3048 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:36:03 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:39290
I20260809 04:36:03.996665 3250 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:36:03 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:45748
I20260809 04:36:03.997225 3420 tablet_replica.cc:330] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: stopping tablet replica
I20260809 04:36:03.997452 3417 tablet_replica.cc:330] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: stopping tablet replica
I20260809 04:36:03.997674 3420 raft_consensus.cc:2229] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:36:03.997866 3418 tablet_replica.cc:330] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436: stopping tablet replica
I20260809 04:36:03.998028 3420 raft_consensus.cc:2258] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:36:03.998051 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 2748
I20260809 04:36:03.998138 3418 raft_consensus.cc:2229] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 37de81dd1e5840d1986d534597d96436 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:36:04.003361 3417 raft_consensus.cc:2229] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:36:04.003729 3417 raft_consensus.cc:2258] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:36:04.004488 3420 ts_tablet_manager.cc:1905] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:36:04.005206 3417 ts_tablet_manager.cc:1905] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:36:04.007380 3420 ts_tablet_manager.cc:1918] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:36:04.007496 3420 log.cc:1196] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/6dc2a3a33f1b4007b6f843f75a1b6e5c
I20260809 04:36:04.007843 3420 ts_tablet_manager.cc:1939] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 9320af207b364f2793f0729f989bed87: Deleting consensus metadata
I20260809 04:36:04.007826 3417 ts_tablet_manager.cc:1918] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:36:04.007922 3417 log.cc:1196] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/6dc2a3a33f1b4007b6f843f75a1b6e5c
I20260809 04:36:04.008234 3417 ts_tablet_manager.cc:1939] T 6dc2a3a33f1b4007b6f843f75a1b6e5c P 4587816bdca146e3b74f2185f5996828: Deleting consensus metadata
I20260809 04:36:04.009285 2617 catalog_manager.cc:4841] TS 9320af207b364f2793f0729f989bed87 (127.19.92.3:35573): tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c (table test-table [id=47a0846d38b04951a43a0fd13d0a5c2e]) successfully deleted
I20260809 04:36:04.009443 2612 catalog_manager.cc:4841] TS 4587816bdca146e3b74f2185f5996828 (127.19.92.2:41613): tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c (table test-table [id=47a0846d38b04951a43a0fd13d0a5c2e]) successfully deleted
W20260809 04:36:04.009603 2617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:35615 (error 108)
I20260809 04:36:04.009634 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 2955
I20260809 04:36:04.027845 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:04.027464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:35615 (local address 127.0.0.1:47256)
0809 04:36:04.027641 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:04.027739 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 04:36:04.028064 2617 catalog_manager.cc:4584] TS 37de81dd1e5840d1986d534597d96436 (127.19.92.1:35615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6dc2a3a33f1b4007b6f843f75a1b6e5c: Network error: Client connection negotiation failed: client connection to 127.19.92.1:35615: connect: Connection refused (error 111)
I20260809 04:36:04.032143 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 3162
I20260809 04:36:04.051335 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 2584
2026-08-09T04:36:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2416 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:36:04 dist-test-slave-rmhg krb5kdc[3427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:36:04 dist-test-slave-rmhg krb5kdc[3427](info): set up 2 sockets
Aug 09 04:36:04 dist-test-slave-rmhg krb5kdc[3427](info): commencing operation
krb5kdc: starting...
W20260809 04:36:05.179754 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:36:05 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:36:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/rangeradmin_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/rangeradmin_127.19.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/rangerlookup_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/rangerlookup_127.19.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/HTTP_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/HTTP_127.19.92.20@KRBTEST.COM.keytab.
I20260809 04:36:05.359823 19824 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWTI4ci/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/postgres -l logfile start
2026-08-09 04:36:07.817 UTC [3489] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 04:36:07.817 UTC [3489] LOG: listening on IPv4 address "127.19.92.20", port 39021
2026-08-09 04:36:07.821 UTC [3489] LOG: listening on Unix socket "/tmp/.s.PGSQL.39021"
2026-08-09 04:36:07.849 UTC [3492] LOG: database system was shut down at 2026-08-09 04:36:06 UTC
2026-08-09 04:36:07.852 UTC [3489] LOG: database system is ready to accept connections
I20260809 04:36:08.881251 19824 mini_postgres.cc:96] Postgres bound to 39021
2026-08-09 04:36:08.889 UTC [3509] FATAL: database "slave" does not exist
127.19.92.20:39021 - accepting connections
I20260809 04:36:08.890249 19824 mini_ranger.cc:161] Starting Ranger...
I20260809 04:36:08.906999 19824 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 04:36:09.350943 19824 mini_ranger.cc:87] Created ranger Postgres database
I20260809 04:36:09.351035 19824 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-admin
2026-08-09 04:36:09,693 [I] DB FLAVOR :POSTGRES
2026-08-09 04:36:09,694 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 04:36:09,694 [I] Checking connection..
2026-08-09 04:36:09,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 04:36:09,968 [I] Checking connection passed.
2026-08-09 04:36:09,968 [I] --------- Verifying version history table ---------
2026-08-09 04:36:09,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:36:10,209 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 04:36:10,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:36:10,476 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 04:36:10,476 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 04:36:10,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 04:36:10.739 UTC [3614] WARNING: there is no transaction in progress
2026-08-09 04:36:10,747 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 04:36:10,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:36:10,985 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 04:36:10,985 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 04:36:10,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 04:36:11,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 04:36:11,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-09 04:36:11,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 04:36:12,028 [I] Table x_portal_user does not exist in database ranger
2026-08-09 04:36:12,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 04:36:12,296 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 04:36:12,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 04:36:12,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 04:36:12,793 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 04:36:12,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 04:36:13.776 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.792 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.808 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.823 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.842 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.862 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.928 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.936 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.951 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.961 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.972 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.985 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:13.995 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:14.001 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:14.008 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:14.015 UTC [3778] WARNING: there is no transaction in progress
2026-08-09 04:36:14,650 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 04:36:14,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 04:36:14,906 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 04:36:14,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 04:36:15,174 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 04:36:15,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 04:36:15,434 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 04:36:15,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 04:36:15,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 04:36:15,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-09 04:36:16,168 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 04:36:16,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 04:36:16,422 [I] DB_PATCHES have already been applied
I20260809 04:36:16.428846 19824 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWTI4ci/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 04:36:16.428938 19824 mini_ranger.cc:193] Using host: 127.19.92.20
I20260809 04:36:16.433646 19824 mini_ranger.cc:227] Ranger admin URL: http://127.19.92.20:47199
Aug 09, 2026 4:36:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:36:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:36:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 4:36:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 4:36:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 4:36:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 4:36:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 04:36:19.130263 19824 mini_ranger.cc:160] Time spent starting Ranger: real 10.240s user 0.005s sys 0.010s
Aug 09, 2026 4:36:40 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 4:36:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 4:36:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 4:36:40 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 4:36:41 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 04:36:41.811336 19824 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/rangerkms_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/rangerkms_127.19.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.92.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.92.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.92.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/HTTP_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.92.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/HTTP_127.19.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 04:36:41 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250201, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 04:36:41.915132 19824 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 04:36:41.928138 19824 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 04:36:42.717495 19824 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 04:36:42.717600 19824 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms
I20260809 04:36:42.717705 19824 mini_ranger_kms.cc:227] Using postgres at 127.19.92.20:39021
2026-08-09 04:36:43,490 [I] DB FLAVOR :POSTGRES
2026-08-09 04:36:43,491 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 04:36:43,491 [I] Checking connection
2026-08-09 04:36:43,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 04:36:43,856 [I] connection success
2026-08-09 04:36:43,856 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 04:36:43,856 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 04:36:43,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 04:36:44,189 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 04:36:44,189 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 04:36:44,189 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 04:36:44,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:39021/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 04:36:44,558 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 04:36:45.032467 19824 mini_ranger_kms.cc:310] Created kms service
I20260809 04:36:45.219926 19824 mini_ranger_kms.cc:326] Created kudu user
I20260809 04:36:45.263690 19824 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 04:36:45 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250201, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:36:45.689232 19824 mini_ranger_kms.cc:384] Added ranger policy
I20260809 04:36:45.689410 19824 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 04:36:45.689447 19824 mini_ranger_kms.cc:240] Using host: 127.19.92.20
I20260809 04:36:45.694422 19824 mini_ranger_kms.cc:276] Ranger KMS PID: 4049
I20260809 04:36:45.694581 19824 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.92.20:43397
Aug 09, 2026 4:36:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:36:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 4:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 4:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 4:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260809 04:36:48.219218 19824 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.304s user 0.018s sys 0.709s
I20260809 04:36:48.219357 19824 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 04:36:48.219475 19824 mini_ranger_kms.cc:401] 127.19.92.20:43397/kms/v1/keys
Aug 09 04:36:48 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786250208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.92.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 04:36:48 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.92.20@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
Aug 09 04:36:49 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250208, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.92.20@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
Aug 09 04:36:51 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:36:51.791709 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:35387
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:35387
--ranger_config_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:43397/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:36:51.943257 4127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:51.943542 4127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:51.943583 4127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:51.947682 4127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:36:51.947752 4127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:51.947770 4127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:51.947785 4127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:36:51.947803 4127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:36:51.953428 4127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.92.20:43397/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:35387
--ranger_config_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:35387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:51.954496 4127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:51.955554 4127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:51.964681 4136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:51.964807 4127 server_base.cc:1034] running on GCE node
W20260809 04:36:51.964753 4135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:51.964730 4139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:51.965549 4127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:51.966670 4127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:51.967828 4127 hybrid_clock.cc:648] HybridClock initialized: now 1786250211967815 us; error 29 us; skew 500 ppm
Aug 09 04:36:51 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250211, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:36:51.975828 4127 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:36:51.976819 4127 webserver.cc:460] Webserver started at http://127.19.92.62:35821/ using document root <none> and password file <none>
I20260809 04:36:51.977425 4127 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:51.977489 4127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:51.977694 4127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:36:51 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250211, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:36:52.027496 4127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "6199d1a3ed214114aa4d1a6266f9af54"
format_stamp: "Formatted at 2026-08-09 04:36:52 on dist-test-slave-rmhg"
server_key: "8069ba72c343c0ee070ee8635a873c46"
server_key_iv: "5f51d601b721e353f86cf555f38547e3"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:52.028280 4127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "6199d1a3ed214114aa4d1a6266f9af54"
format_stamp: "Formatted at 2026-08-09 04:36:52 on dist-test-slave-rmhg"
server_key: "8069ba72c343c0ee070ee8635a873c46"
server_key_iv: "5f51d601b721e353f86cf555f38547e3"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:52.033316 4127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 04:36:52.051419 4155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:52.053140 4127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:36:52.053318 4127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "6199d1a3ed214114aa4d1a6266f9af54"
format_stamp: "Formatted at 2026-08-09 04:36:52 on dist-test-slave-rmhg"
server_key: "8069ba72c343c0ee070ee8635a873c46"
server_key_iv: "5f51d601b721e353f86cf555f38547e3"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:52.053457 4127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:52.071676 4127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:52.074335 4127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:52.074501 4127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:52.087666 4127 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:35387
I20260809 04:36:52.087832 4234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:35387 every 8 connection(s)
I20260809 04:36:52.088817 4127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:36:52.091641 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 4127
I20260809 04:36:52.091775 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:36:52.092056 19824 external_mini_cluster.cc:1428] Setting key aa439058e969eac42d24c24970ad166c
I20260809 04:36:52.092105 4236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:52.100898 4236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54: Bootstrap starting.
Aug 09 04:36:52 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:36:52.104156 4236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:52.105342 4236 log.cc:824] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:52.107909 4236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54: No bootstrap required, opened a new log
I20260809 04:36:52.108752 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:52.093594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.19.92.62:35387)
0809 04:36:52.094092 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:52.094111 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:52.094148 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:36:52.095291 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:52.095313 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:52.095676 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:52.095978 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:52.097019 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:52.098354 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:52.099626 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:52.100122 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:52.100240 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:52.103375 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:52.103382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:52.103397 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:52.103434 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:52.105910 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:52.106371 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:52.106376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:52.106381 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:52.106494 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:52.106771 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:52.106773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:52.106774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:52.107193 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:52.107336 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:52.107648 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 04:36:52.107913 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":195,"threads_started":1}
I20260809 04:36:52.112092 4236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6199d1a3ed214114aa4d1a6266f9af54" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35387 } }
I20260809 04:36:52.112349 4236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:52.112424 4236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6199d1a3ed214114aa4d1a6266f9af54, State: Initialized, Role: FOLLOWER
I20260809 04:36:52.112893 4236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6199d1a3ed214114aa4d1a6266f9af54" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35387 } }
I20260809 04:36:52.113047 4236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:36:52.113216 4236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:36:52.113332 4236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:52.114609 4236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6199d1a3ed214114aa4d1a6266f9af54" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35387 } }
I20260809 04:36:52.114964 4236 leader_election.cc:302] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6199d1a3ed214114aa4d1a6266f9af54; no voters:
I20260809 04:36:52.115427 4236 leader_election.cc:288] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:36:52.115636 4244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:36:52.116003 4244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [term 1 LEADER]: Becoming Leader. State: Replica: 6199d1a3ed214114aa4d1a6266f9af54, State: Running, Role: LEADER
I20260809 04:36:52.116334 4244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6199d1a3ed214114aa4d1a6266f9af54" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35387 } }
I20260809 04:36:52.116876 4236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:36:52.118407 4247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6199d1a3ed214114aa4d1a6266f9af54. Latest consensus state: current_term: 1 leader_uuid: "6199d1a3ed214114aa4d1a6266f9af54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6199d1a3ed214114aa4d1a6266f9af54" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35387 } } }
I20260809 04:36:52.118453 4245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6199d1a3ed214114aa4d1a6266f9af54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6199d1a3ed214114aa4d1a6266f9af54" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 35387 } } }
I20260809 04:36:52.118599 4247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [sys.catalog]: This master's current role is: LEADER
I20260809 04:36:52.118599 4245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54 [sys.catalog]: This master's current role is: LEADER
I20260809 04:36:52.119184 4258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:36:52.122746 4258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:36:52.123260 4236 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 04:36:52.133441 4258 catalog_manager.cc:1347] Generated new cluster ID: 1382b10102d04b32970936c7160e0bd6
I20260809 04:36:52.133530 4258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:36:52.161343 4258 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:36:52.162891 4258 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:36:52.171847 4258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54: Generated new TSK 0
I20260809 04:36:52.172732 4258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:36:53.018827 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:43397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35387
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:36:53.185825 4306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:53.186131 4306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:53.186198 4306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:53.190460 4306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:53.190529 4306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:53.190616 4306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:36:53.196300 4306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.92.20:43397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:53.197698 4306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:53.199169 4306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:53.212329 4319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:53.212371 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:53.212419 4325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:53.212872 4306 server_base.cc:1034] running on GCE node
I20260809 04:36:53.213378 4306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:53.214094 4306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:53.215263 4306 hybrid_clock.cc:648] HybridClock initialized: now 1786250213215209 us; error 74 us; skew 500 ppm
Aug 09 04:36:53 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:36:53.218583 4306 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:36:53.219617 4306 webserver.cc:460] Webserver started at http://127.19.92.1:33809/ using document root <none> and password file <none>
I20260809 04:36:53.220191 4306 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:53.220259 4306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:53.220487 4306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:36:53 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:36:53.238899 4306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "ed3dc12245544e198ea91caf734ad63a"
format_stamp: "Formatted at 2026-08-09 04:36:53 on dist-test-slave-rmhg"
server_key: "dd5f8348b0c70be8e71c0fb858c55808"
server_key_iv: "4decad6c4e2056c77a7b6aa2754145c6"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:53.239593 4306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "ed3dc12245544e198ea91caf734ad63a"
format_stamp: "Formatted at 2026-08-09 04:36:53 on dist-test-slave-rmhg"
server_key: "dd5f8348b0c70be8e71c0fb858c55808"
server_key_iv: "4decad6c4e2056c77a7b6aa2754145c6"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:53.246632 4306 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260809 04:36:53.265359 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:53.267499 4306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260809 04:36:53.267757 4306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "ed3dc12245544e198ea91caf734ad63a"
format_stamp: "Formatted at 2026-08-09 04:36:53 on dist-test-slave-rmhg"
server_key: "dd5f8348b0c70be8e71c0fb858c55808"
server_key_iv: "4decad6c4e2056c77a7b6aa2754145c6"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:53.268033 4306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:53.299937 4306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:53.304139 4306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:53.304340 4306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:53.305275 4306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:53.306344 4306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:53.306394 4306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:53.306435 4306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:53.306463 4306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:53.329290 4306 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:41589
I20260809 04:36:53.329550 4518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:41589 every 8 connection(s)
I20260809 04:36:53.330729 4306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:36:53.339030 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 4306
I20260809 04:36:53.339196 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:36:53.339526 19824 external_mini_cluster.cc:1428] Setting key f775a9629aed21c2cd36259272ef7222
Aug 09 04:36:53 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:36:53.360173 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:53.334507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35387 (local address 127.19.92.1:60185)
0809 04:36:53.335260 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:53.335328 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:53.336780 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:53.337846 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:53.337858 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:53.338270 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:53.339103 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:53.339126 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:53.347261 (+ 8135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:53.347265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:53.349080 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:53.349100 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:53.349900 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:53.349905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:53.350224 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:53.351101 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:36:53.351126 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:53.353420 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:53.355811 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:53.355819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:53.355839 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:53.356236 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:53.356583 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:53.356587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:53.356589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:53.356796 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:53.357535 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:53.357544 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:53.358114 (+ 570us) client_negotiation.cc:769] Sending connection context
0809 04:36:53.358439 (+ 325us) client_negotiation.cc:240] Negotiation successful
0809 04:36:53.358865 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":195,"threads_started":1}
I20260809 04:36:53.361797 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:53.335655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:60185 (local address 127.19.92.62:35387)
0809 04:36:53.337320 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:53.337326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:53.337343 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:36:53.337393 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:53.337397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:53.337459 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:53.339401 (+ 1942us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:53.339556 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:53.347001 (+ 7445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:53.349314 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:53.349685 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:53.350722 (+ 1037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:53.353647 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:53.353651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:53.353654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:53.353688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:53.355657 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:53.356371 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:53.356374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:53.356376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:53.356475 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:53.356980 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:53.356983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:53.356985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:53.357321 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:53.361344 (+ 4023us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:53.361455 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 04:36:53.361616 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1565,"thread_start_us":218,"threads_started":1}
I20260809 04:36:53.363221 4519 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35387
I20260809 04:36:53.363511 4519 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:53.364182 4519 heartbeater.cc:511] Master 127.19.92.62:35387 requested a full tablet report, sending...
I20260809 04:36:53.366189 4177 ts_manager.cc:194] Registered new tserver with Master: ed3dc12245544e198ea91caf734ad63a (127.19.92.1:41589)
I20260809 04:36:53.367862 4177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:60185
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:36:53.404709 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:43397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35387
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:36:53 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 04:36:53.609246 4533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:53.609544 4533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:53.609591 4533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:53.614934 4533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:53.615026 4533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:53.615113 4533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:36:53.621294 4533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.92.20:43397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:53.622431 4533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:53.623541 4533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:53.638723 4542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:53.638892 4544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:53.639154 4541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:53.639211 4533 server_base.cc:1034] running on GCE node
I20260809 04:36:53.639923 4533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:53.640740 4533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:53.644053 4533 hybrid_clock.cc:648] HybridClock initialized: now 1786250213643974 us; error 95 us; skew 500 ppm
Aug 09 04:36:53 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:36:53.647656 4533 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:36:53.648818 4533 webserver.cc:460] Webserver started at http://127.19.92.2:38425/ using document root <none> and password file <none>
I20260809 04:36:53.649624 4533 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:53.649741 4533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:53.650055 4533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:36:53 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:36:53.672817 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "e50947b2e6e54c75b938849c84aa5fde"
format_stamp: "Formatted at 2026-08-09 04:36:53 on dist-test-slave-rmhg"
server_key: "377594146859267229105cff9ad97365"
server_key_iv: "0ed3956013c294545976299446c32a38"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:53.673532 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "e50947b2e6e54c75b938849c84aa5fde"
format_stamp: "Formatted at 2026-08-09 04:36:53 on dist-test-slave-rmhg"
server_key: "377594146859267229105cff9ad97365"
server_key_iv: "0ed3956013c294545976299446c32a38"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:53.677992 4533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:36:53.703701 4552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:53.705636 4533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 04:36:53.705780 4533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "e50947b2e6e54c75b938849c84aa5fde"
format_stamp: "Formatted at 2026-08-09 04:36:53 on dist-test-slave-rmhg"
server_key: "377594146859267229105cff9ad97365"
server_key_iv: "0ed3956013c294545976299446c32a38"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:53.705927 4533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:53.725687 4533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:53.731722 4533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:53.731997 4533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:53.732793 4533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:53.734125 4533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:53.734238 4533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:53.734341 4533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:53.734414 4533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:53.759706 4533 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:41121
I20260809 04:36:53.761348 4533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:36:53.765405 4727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:41121 every 8 connection(s)
I20260809 04:36:53.771193 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 4533
I20260809 04:36:53.771338 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:36:53.771694 19824 external_mini_cluster.cc:1428] Setting key 1d5fbe3e42730c58033a76d5b0f3594f
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Aug 09 04:36:53 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:36:53.806919 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:53.768899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:37495 (local address 127.19.92.62:35387)
0809 04:36:53.769191 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:53.769196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:53.781239 (+ 12043us) server_negotiation.cc:407] Connection header received
0809 04:36:53.793742 (+ 12503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:53.793746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:53.793823 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:53.793880 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:53.796220 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:53.797051 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:53.799028 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:53.799277 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:53.799393 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:53.802616 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:53.802621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:53.802624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:53.802661 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:53.804378 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:53.805026 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:53.805029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:53.805031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:53.805122 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:53.805573 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:53.805576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:53.805578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:53.805810 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:53.805908 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:53.806610 (+ 702us) server_negotiation.cc:299] Negotiation successful
0809 04:36:53.806748 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:36:53.807976 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:53.769518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35387 (local address 127.19.92.2:37495)
0809 04:36:53.780957 (+ 11439us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:53.781059 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:53.793345 (+ 12286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:53.794473 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:53.794485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:53.795115 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:53.796040 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:53.796062 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:53.797287 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:53.797292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:53.798843 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:53.798858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:53.799495 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:53.799501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:53.799742 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:53.800469 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:36:53.800494 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:53.802288 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:53.804540 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:53.804547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:53.804567 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:53.804897 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:53.805245 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:53.805249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:53.805251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:53.805462 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:53.805927 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:53.805934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:53.806314 (+ 380us) client_negotiation.cc:769] Sending connection context
0809 04:36:53.806549 (+ 235us) client_negotiation.cc:240] Negotiation successful
0809 04:36:53.806925 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11052,"thread_start_us":215,"threads_started":1}
I20260809 04:36:53.809352 4734 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35387
I20260809 04:36:53.809626 4734 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:53.810248 4734 heartbeater.cc:511] Master 127.19.92.62:35387 requested a full tablet report, sending...
I20260809 04:36:53.811519 4177 ts_manager.cc:194] Registered new tserver with Master: e50947b2e6e54c75b938849c84aa5fde (127.19.92.2:41121)
I20260809 04:36:53.812326 4177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:37495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:36:53.828469 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:43397/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35387
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:36:54.041028 4740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:36:54.041618 4740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:36:54.041741 4740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:36:54.047668 4740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:36:54.047883 4740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:36:54.048018 4740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:36:54.055830 4740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.92.20:43397/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:36:54.057461 4740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:36:54.058811 4740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:36:54.082376 4748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:36:54.082854 4751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:54.084676 4740 server_base.cc:1034] running on GCE node
W20260809 04:36:54.085009 4746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:36:54.085728 4740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:36:54.086694 4740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:36:54.088054 4740 hybrid_clock.cc:648] HybridClock initialized: now 1786250214087982 us; error 90 us; skew 500 ppm
Aug 09 04:36:54 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
Aug 09 04:36:54 dist-test-slave-rmhg krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:36:54.093106 4740 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:36:54.094349 4740 webserver.cc:460] Webserver started at http://127.19.92.3:45481/ using document root <none> and password file <none>
I20260809 04:36:54.095145 4740 fs_manager.cc:359] Metadata directory not provided
I20260809 04:36:54.095292 4740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:36:54.095604 4740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:36:54 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:36:54.113044 4740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "87e24a8fa0d3448fb13f682b73ea3044"
format_stamp: "Formatted at 2026-08-09 04:36:54 on dist-test-slave-rmhg"
server_key: "7a68a24bf30920a1eec3617fc1da2e66"
server_key_iv: "e44cb91da5a582d2547959e32055bad3"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:54.113876 4740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "87e24a8fa0d3448fb13f682b73ea3044"
format_stamp: "Formatted at 2026-08-09 04:36:54 on dist-test-slave-rmhg"
server_key: "7a68a24bf30920a1eec3617fc1da2e66"
server_key_iv: "e44cb91da5a582d2547959e32055bad3"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:54.118597 4740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:36:54.152129 4760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:54.154384 4740 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260809 04:36:54.154582 4740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "87e24a8fa0d3448fb13f682b73ea3044"
format_stamp: "Formatted at 2026-08-09 04:36:54 on dist-test-slave-rmhg"
server_key: "7a68a24bf30920a1eec3617fc1da2e66"
server_key_iv: "e44cb91da5a582d2547959e32055bad3"
server_key_version: "kuduclusterkey@0"
I20260809 04:36:54.154738 4740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:36:54.175078 4740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:36:54.179601 4740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:36:54.179992 4740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:36:54.181295 4740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:36:54.182642 4740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:36:54.182757 4740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:54.182865 4740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:36:54.182936 4740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:36:54.211297 4740 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:37179
I20260809 04:36:54.212728 4740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:36:54.213821 4946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:37179 every 8 connection(s)
I20260809 04:36:54.221007 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 4740
I20260809 04:36:54.221184 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:36:54.221654 19824 external_mini_cluster.cc:1428] Setting key 50428861d9230a8bc4e94b55ebf0044c
Aug 09 04:36:54 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250214, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:36:54.241076 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.218990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:46115 (local address 127.19.92.62:35387)
0809 04:36:54.219125 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:54.219130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:54.219721 (+ 591us) server_negotiation.cc:407] Connection header received
0809 04:36:54.221519 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:54.221524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:54.221586 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:54.221640 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:54.223608 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.224456 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.226331 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.226678 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.226748 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.236332 (+ 9584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:54.236336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:54.236339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:54.236374 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:54.238257 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:54.239020 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:54.239024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:54.239026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:54.239124 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:54.239587 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:54.239590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:54.239592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:54.239844 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:54.239924 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:54.240746 (+ 822us) server_negotiation.cc:299] Negotiation successful
0809 04:36:54.240897 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:36:54.243253 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.217596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:35387 (local address 127.19.92.3:46115)
0809 04:36:54.219467 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:54.219530 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:54.221235 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:54.222117 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:54.222127 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:54.222692 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:36:54.223441 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.223464 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.224618 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.224621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.226193 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.226205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.226965 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.226969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.227212 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.231630 (+ 4418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:36:54.231656 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:36:54.236111 (+ 4455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:36:54.238448 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:54.238456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:54.238480 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:54.238883 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:54.239231 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:36:54.239234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:36:54.239236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:36:54.239473 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:36:54.240003 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:36:54.240011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:36:54.240430 (+ 419us) client_negotiation.cc:769] Sending connection context
0809 04:36:54.241645 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0809 04:36:54.242051 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"thread_start_us":254,"threads_started":1}
I20260809 04:36:54.245241 4947 heartbeater.cc:347] Connected to a master server at 127.19.92.62:35387
I20260809 04:36:54.245554 4947 heartbeater.cc:464] Registering TS with master...
I20260809 04:36:54.246194 4947 heartbeater.cc:511] Master 127.19.92.62:35387 requested a full tablet report, sending...
I20260809 04:36:54.247337 4176 ts_manager.cc:194] Registered new tserver with Master: 87e24a8fa0d3448fb13f682b73ea3044 (127.19.92.3:37179)
I20260809 04:36:54.248034 4176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:46115
I20260809 04:36:54.250792 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:36:54.264765 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.253970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.19.92.62:35387)
0809 04:36:54.254109 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:54.254113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:54.254966 (+ 853us) server_negotiation.cc:407] Connection header received
0809 04:36:54.255188 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:54.255192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:54.255249 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:54.255303 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:36:54.256346 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.257196 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.258500 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.258849 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.258934 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.260986 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:36:54.260990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:36:54.260993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:36:54.261026 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:36:54.262845 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:54.263359 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:54.263363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:54.263365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:54.263467 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:36:54.263773 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:36:54.263776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:36:54.263778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:36:54.263997 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:36:54.264085 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:54.264457 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 04:36:54.264606 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:36:54.268560 4176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:36:54.272922 4176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:36:54.301497 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.290994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:41121 (local address 127.0.0.1:48130)
0809 04:36:54.293824 (+ 2830us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:54.293837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:54.294047 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:54.295543 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:54.295546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:54.295566 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:54.295668 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.295674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.297319 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.297323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.298553 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.298565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.300865 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.300869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.301007 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.301053 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 04:36:54.301193 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 04:36:54.301329 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2766,"thread_start_us":98,"threads_started":1}
I20260809 04:36:54.302372 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.288979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:37179 (local address 127.0.0.1:56904)
0809 04:36:54.291581 (+ 2602us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:54.291642 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:54.291864 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:54.293830 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:54.293837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:54.293865 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:54.294019 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.294028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.295734 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.295737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.297298 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.297311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.301960 (+ 4649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.301964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.302061 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.302090 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 04:36:54.302147 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 04:36:54.302223 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2383,"thread_start_us":303,"threads_started":1}
I20260809 04:36:54.302433 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.291436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.19.92.2:41121)
0809 04:36:54.292381 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:54.292402 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:54.294144 (+ 1742us) server_negotiation.cc:407] Connection header received
0809 04:36:54.294229 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:54.294234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:54.294440 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:54.295793 (+ 1353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:54.295962 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.297183 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.299621 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.300521 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.300631 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.300799 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:54.302175 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0809 04:36:54.302290 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":168,"threads_started":1}
I20260809 04:36:54.303323 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.289501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.19.92.3:37179)
0809 04:36:54.292951 (+ 3450us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:54.293004 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:54.293030 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:36:54.293109 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:54.293114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:54.293357 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:54.293570 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:54.294185 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.295575 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.297591 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.298618 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.302818 (+ 4200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.303001 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:54.303056 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 04:36:54.303150 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3327,"thread_start_us":291,"threads_started":1}
I20260809 04:36:54.306677 4835 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5c3e6e5727441c843f10ffd3875572 (DEFAULT_TABLE table=test-table [id=b9f2b64fdf384df8bd6b9c77350e3b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:36:54.306658 4608 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5c3e6e5727441c843f10ffd3875572 (DEFAULT_TABLE table=test-table [id=b9f2b64fdf384df8bd6b9c77350e3b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:36:54.307925 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5c3e6e5727441c843f10ffd3875572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:54.307929 4608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5c3e6e5727441c843f10ffd3875572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:54.323783 4977 tablet_bootstrap.cc:492] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Bootstrap starting.
I20260809 04:36:54.327358 4977 tablet_bootstrap.cc:654] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:54.328236 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.290992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.19.92.1:41589)
0809 04:36:54.292179 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:54.292188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:54.292673 (+ 485us) server_negotiation.cc:407] Connection header received
0809 04:36:54.292776 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:54.292783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:54.293039 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:54.293624 (+ 585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:54.299124 (+ 5500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.300348 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.313283 (+ 12935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.314285 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.314423 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.314599 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:54.327853 (+ 13254us) server_negotiation.cc:299] Negotiation successful
0809 04:36:54.328028 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":102,"threads_started":1}
I20260809 04:36:54.328934 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.290581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:41589 (local address 127.0.0.1:44202)
0809 04:36:54.292464 (+ 1883us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:54.292479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:54.292668 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:54.298743 (+ 6075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:54.298746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:54.298767 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:54.298961 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.298970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.300522 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.300525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.301777 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.301790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.316333 (+ 14543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.316338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.327579 (+ 11241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.327628 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 04:36:54.328669 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0809 04:36:54.328788 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1811,"thread_start_us":191,"threads_started":1}
I20260809 04:36:54.328989 4977 log.cc:824] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:54.330044 4978 tablet_bootstrap.cc:492] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Bootstrap starting.
I20260809 04:36:54.332674 4411 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5c3e6e5727441c843f10ffd3875572 (DEFAULT_TABLE table=test-table [id=b9f2b64fdf384df8bd6b9c77350e3b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:36:54.334259 4411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5c3e6e5727441c843f10ffd3875572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:36:54.335738 4977 tablet_bootstrap.cc:492] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: No bootstrap required, opened a new log
I20260809 04:36:54.335973 4977 ts_tablet_manager.cc:1397] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20260809 04:36:54.342026 4977 raft_consensus.cc:348] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.342327 4977 raft_consensus.cc:374] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:54.342384 4977 raft_consensus.cc:729] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50947b2e6e54c75b938849c84aa5fde, State: Initialized, Role: FOLLOWER
I20260809 04:36:54.344265 4978 tablet_bootstrap.cc:654] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:54.345791 4978 log.cc:824] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:54.346678 4977 consensus_queue.cc:260] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.348017 4977 ts_tablet_manager.cc:1428] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Time spent starting tablet: real 0.012s user 0.004s sys 0.005s
I20260809 04:36:54.348398 4734 heartbeater.cc:503] Master 127.19.92.62:35387 was elected leader, sending a full tablet report...
I20260809 04:36:54.352488 4978 tablet_bootstrap.cc:492] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: No bootstrap required, opened a new log
I20260809 04:36:54.352720 4978 ts_tablet_manager.cc:1397] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260809 04:36:54.352705 4980 tablet_bootstrap.cc:492] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a: Bootstrap starting.
I20260809 04:36:54.356089 4980 tablet_bootstrap.cc:654] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a: Neither blocks nor log segments found. Creating new log.
I20260809 04:36:54.359910 4978 raft_consensus.cc:348] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.360280 4978 raft_consensus.cc:374] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:54.360352 4978 raft_consensus.cc:729] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e24a8fa0d3448fb13f682b73ea3044, State: Initialized, Role: FOLLOWER
I20260809 04:36:54.360956 4978 consensus_queue.cc:260] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.362401 4978 ts_tablet_manager.cc:1428] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260809 04:36:54.362816 4947 heartbeater.cc:503] Master 127.19.92.62:35387 was elected leader, sending a full tablet report...
I20260809 04:36:54.364779 4980 log.cc:824] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:36:54.368748 4980 tablet_bootstrap.cc:492] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a: No bootstrap required, opened a new log
I20260809 04:36:54.369000 4980 ts_tablet_manager.cc:1397] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260809 04:36:54.371189 4519 heartbeater.cc:503] Master 127.19.92.62:35387 was elected leader, sending a full tablet report...
I20260809 04:36:54.376358 4980 raft_consensus.cc:348] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.376708 4980 raft_consensus.cc:374] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:36:54.376776 4980 raft_consensus.cc:729] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3dc12245544e198ea91caf734ad63a, State: Initialized, Role: FOLLOWER
I20260809 04:36:54.377313 4980 consensus_queue.cc:260] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.378742 4980 ts_tablet_manager.cc:1428] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20260809 04:36:54.465236 4949 tablet.cc:2369] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 04:36:54 dist-test-slave-rmhg krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250213, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
W20260809 04:36:54.519490 4735 tablet.cc:2369] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:36:54.541555 4983 raft_consensus.cc:482] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:36:54.541813 4983 raft_consensus.cc:504] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.543467 4983 leader_election.cc:288] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3dc12245544e198ea91caf734ad63a (127.19.92.1:41589), e50947b2e6e54c75b938849c84aa5fde (127.19.92.2:41121)
I20260809 04:36:54.548838 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.544628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:48335 (local address 127.19.92.1:41589)
0809 04:36:54.544884 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:54.544888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:54.544904 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:36:54.544962 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:54.544965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:54.545015 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:54.545098 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:54.545599 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.546366 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.547566 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.548133 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.548234 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.548315 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:54.548625 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 04:36:54.548707 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260809 04:36:54.548841 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.543845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:41589 (local address 127.19.92.3:48335)
0809 04:36:54.544608 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:54.544624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:54.544733 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:54.545113 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:54.545116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:54.545139 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:54.545294 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.545303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.546520 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.546523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.547436 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.547446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.548281 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.548286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.548485 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.548515 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 04:36:54.548605 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 04:36:54.548691 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673}
I20260809 04:36:54.550117 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5c3e6e5727441c843f10ffd3875572" candidate_uuid: "87e24a8fa0d3448fb13f682b73ea3044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3dc12245544e198ea91caf734ad63a" is_pre_election: true
I20260809 04:36:54.550565 4443 raft_consensus.cc:2454] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e24a8fa0d3448fb13f682b73ea3044 in term 0.
I20260809 04:36:54.551327 4766 leader_election.cc:302] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e24a8fa0d3448fb13f682b73ea3044, ed3dc12245544e198ea91caf734ad63a; no voters:
I20260809 04:36:54.551734 4983 raft_consensus.cc:2781] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:36:54.551829 4983 raft_consensus.cc:482] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:36:54.551877 4983 raft_consensus.cc:3037] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:54.552024 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.545056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:35435 (local address 127.19.92.2:41121)
0809 04:36:54.545497 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:54.545502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:54.547728 (+ 2226us) server_negotiation.cc:407] Connection header received
0809 04:36:54.547913 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:54.547916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:54.547965 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:54.548024 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:36:54.548480 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.549144 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.550582 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.551184 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.551286 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.551354 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:54.551759 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 04:36:54.551867 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260809 04:36:54.553185 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.544135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:41121 (local address 127.19.92.3:35435)
0809 04:36:54.547579 (+ 3444us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:54.547593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:36:54.547803 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:36:54.548166 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:36:54.548169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:36:54.548188 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:36:54.548327 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.548333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.549514 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.549517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.550461 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:36:54.550473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.551455 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.551459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:36:54.551604 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.551632 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 04:36:54.552899 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0809 04:36:54.553000 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":117,"threads_started":1}
I20260809 04:36:54.553341 4983 raft_consensus.cc:504] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.553694 4983 leader_election.cc:288] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [CANDIDATE]: Term 1 election: Requested vote from peers ed3dc12245544e198ea91caf734ad63a (127.19.92.1:41589), e50947b2e6e54c75b938849c84aa5fde (127.19.92.2:41121)
I20260809 04:36:54.554351 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5c3e6e5727441c843f10ffd3875572" candidate_uuid: "87e24a8fa0d3448fb13f682b73ea3044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3dc12245544e198ea91caf734ad63a"
I20260809 04:36:54.554526 4443 raft_consensus.cc:3037] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:54.554594 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5c3e6e5727441c843f10ffd3875572" candidate_uuid: "87e24a8fa0d3448fb13f682b73ea3044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50947b2e6e54c75b938849c84aa5fde" is_pre_election: true
I20260809 04:36:54.554859 4649 raft_consensus.cc:2454] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e24a8fa0d3448fb13f682b73ea3044 in term 0.
I20260809 04:36:54.555187 4648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5c3e6e5727441c843f10ffd3875572" candidate_uuid: "87e24a8fa0d3448fb13f682b73ea3044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50947b2e6e54c75b938849c84aa5fde"
I20260809 04:36:54.555299 4648 raft_consensus.cc:3037] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:36:54.556432 4648 raft_consensus.cc:2454] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e24a8fa0d3448fb13f682b73ea3044 in term 1.
I20260809 04:36:54.556721 4443 raft_consensus.cc:2454] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e24a8fa0d3448fb13f682b73ea3044 in term 1.
I20260809 04:36:54.556809 4765 leader_election.cc:302] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e24a8fa0d3448fb13f682b73ea3044, e50947b2e6e54c75b938849c84aa5fde; no voters:
I20260809 04:36:54.557183 4983 raft_consensus.cc:2781] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:36:54.557605 4983 raft_consensus.cc:686] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 1 LEADER]: Becoming Leader. State: Replica: 87e24a8fa0d3448fb13f682b73ea3044, State: Running, Role: LEADER
I20260809 04:36:54.558019 4983 consensus_queue.cc:237] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } }
I20260809 04:36:54.562103 4177 catalog_manager.cc:5495] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e24a8fa0d3448fb13f682b73ea3044 (127.19.92.3). New cstate: current_term: 1 leader_uuid: "87e24a8fa0d3448fb13f682b73ea3044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e24a8fa0d3448fb13f682b73ea3044" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 37179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 } health_report { overall_health: UNKNOWN } } }
W20260809 04:36:54.584801 4522 tablet.cc:2369] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:36:54.607558 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.601941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.19.92.3:37179)
0809 04:36:54.602078 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 04:36:54.602083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:36:54.602170 (+ 87us) server_negotiation.cc:407] Connection header received
0809 04:36:54.602363 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:36:54.602367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:36:54.602487 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:36:54.602633 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:36:54.603113 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.603916 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.605225 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:36:54.605552 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:36:54.605650 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:36:54.606167 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:36:54.606804 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:36:54.606948 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 04:36:54.607330 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 04:36:54.607402 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:36:54.628542 4648 raft_consensus.cc:1261] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 1 FOLLOWER]: Refusing update from remote peer 87e24a8fa0d3448fb13f682b73ea3044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:36:54.629539 4983 consensus_queue.cc:1035] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50947b2e6e54c75b938849c84aa5fde" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:36:54.630263 4443 raft_consensus.cc:1261] T 0d5c3e6e5727441c843f10ffd3875572 P ed3dc12245544e198ea91caf734ad63a [term 1 FOLLOWER]: Refusing update from remote peer 87e24a8fa0d3448fb13f682b73ea3044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:36:54.632694 4983 consensus_queue.cc:1035] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3dc12245544e198ea91caf734ad63a" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:36:54.656762 5000 mvcc.cc:205] Tried to move back new op lower bound from 7316480879100407808 to 7316480878833278976. Current Snapshot: MvccSnapshot[applied={T|T < 7316480879100407808}]
I20260809 04:36:54.676517 4177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:36:54.676735 4177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:36:54.679877 4177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6199d1a3ed214114aa4d1a6266f9af54: Sending DeleteTablet for 3 replicas of tablet 0d5c3e6e5727441c843f10ffd3875572
I20260809 04:36:54.681242 4835 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5c3e6e5727441c843f10ffd3875572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:36:54 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:56904
I20260809 04:36:54.681923 4411 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5c3e6e5727441c843f10ffd3875572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:36:54 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:44202
I20260809 04:36:54.683034 4608 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5c3e6e5727441c843f10ffd3875572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:36:54 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:48130
I20260809 04:36:54.684531 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 4306
I20260809 04:36:54.691260 5017 tablet_replica.cc:330] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: stopping tablet replica
I20260809 04:36:54.691720 5017 raft_consensus.cc:2229] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:36:54.692200 5017 raft_consensus.cc:2258] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:36:54.697492 5011 tablet_replica.cc:330] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: stopping tablet replica
I20260809 04:36:54.697945 5011 raft_consensus.cc:2229] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:36:54.698422 5011 raft_consensus.cc:2258] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:36:54.698534 5017 ts_tablet_manager.cc:1905] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:36:54.699955 5011 ts_tablet_manager.cc:1905] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:36:54.701907 5017 ts_tablet_manager.cc:1918] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:36:54.702025 5017 log.cc:1196] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/0d5c3e6e5727441c843f10ffd3875572
I20260809 04:36:54.702327 5017 ts_tablet_manager.cc:1939] T 0d5c3e6e5727441c843f10ffd3875572 P e50947b2e6e54c75b938849c84aa5fde: Deleting consensus metadata
W20260809 04:36:54.702755 4159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:41589 (error 108)
I20260809 04:36:54.703310 5011 ts_tablet_manager.cc:1918] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:36:54.703405 5011 log.cc:1196] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/0d5c3e6e5727441c843f10ffd3875572
I20260809 04:36:54.703655 4158 catalog_manager.cc:4841] TS e50947b2e6e54c75b938849c84aa5fde (127.19.92.2:41121): tablet 0d5c3e6e5727441c843f10ffd3875572 (table test-table [id=b9f2b64fdf384df8bd6b9c77350e3b82]) successfully deleted
I20260809 04:36:54.703714 5011 ts_tablet_manager.cc:1939] T 0d5c3e6e5727441c843f10ffd3875572 P 87e24a8fa0d3448fb13f682b73ea3044: Deleting consensus metadata
I20260809 04:36:54.703994 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 4533
I20260809 04:36:54.717229 4157 catalog_manager.cc:4841] TS 87e24a8fa0d3448fb13f682b73ea3044 (127.19.92.3:37179): tablet 0d5c3e6e5727441c843f10ffd3875572 (table test-table [id=b9f2b64fdf384df8bd6b9c77350e3b82]) successfully deleted
I20260809 04:36:54.720017 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 4740
I20260809 04:36:54.737720 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:36:54.737341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:41589 (local address 127.0.0.1:44218)
0809 04:36:54.737507 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 04:36:54.737609 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:41589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 04:36:54.737962 4159 catalog_manager.cc:4584] TS ed3dc12245544e198ea91caf734ad63a (127.19.92.1:41589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d5c3e6e5727441c843f10ffd3875572: Network error: Client connection negotiation failed: client connection to 127.19.92.1:41589: connect: Connection refused (error 111)
I20260809 04:36:54.744315 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 4127
I20260809 04:36:54.756596 19824 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 04:36:55.103286 19824 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 04:36:55.103473 19824 mini_ranger.cc:66] Stopping Ranger...
I20260809 04:36:55.459021 19824 mini_ranger.cc:68] Stopped Ranger
2026-08-09 04:36:55.459 UTC [3489] LOG: received smart shutdown request
2026-08-09 04:36:55.463 UTC [3489] LOG: background worker "logical replication launcher" (PID 3504) exited with exit code 1
2026-08-09 04:36:55.464 UTC [3493] LOG: shutting down
2026-08-09 04:36:55.625 UTC [3489] LOG: database system is shut down
2026-08-09T04:36:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51669 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:36:55 dist-test-slave-rmhg krb5kdc[5027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:36:55 dist-test-slave-rmhg krb5kdc[5027](info): set up 2 sockets
Aug 09 04:36:55 dist-test-slave-rmhg krb5kdc[5027](info): commencing operation
krb5kdc: starting...
W20260809 04:36:56.868682 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:36:56 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:36:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:36:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/rangeradmin_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/rangeradmin_127.19.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/rangerlookup_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/rangerlookup_127.19.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/HTTP_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/HTTP_127.19.92.20@KRBTEST.COM.keytab.
I20260809 04:36:57.057962 19824 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWTI4ci/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/postgres -l logfile start
2026-08-09 04:36:59.365 UTC [5081] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 04:36:59.365 UTC [5081] LOG: listening on IPv4 address "127.19.92.20", port 48199
2026-08-09 04:36:59.369 UTC [5081] LOG: listening on Unix socket "/tmp/.s.PGSQL.48199"
2026-08-09 04:36:59.389 UTC [5086] LOG: database system was shut down at 2026-08-09 04:36:58 UTC
2026-08-09 04:36:59.392 UTC [5081] LOG: database system is ready to accept connections
I20260809 04:37:00.434880 19824 mini_postgres.cc:96] Postgres bound to 48199
2026-08-09 04:37:00.442 UTC [5095] FATAL: database "slave" does not exist
127.19.92.20:48199 - accepting connections
I20260809 04:37:00.443158 19824 mini_ranger.cc:161] Starting Ranger...
I20260809 04:37:00.459471 19824 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 04:37:00.888556 19824 mini_ranger.cc:87] Created ranger Postgres database
I20260809 04:37:00.888630 19824 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-admin
2026-08-09 04:37:01,179 [I] DB FLAVOR :POSTGRES
2026-08-09 04:37:01,180 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 04:37:01,180 [I] Checking connection..
2026-08-09 04:37:01,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 04:37:01,448 [I] Checking connection passed.
2026-08-09 04:37:01,449 [I] --------- Verifying version history table ---------
2026-08-09 04:37:01,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:37:01,707 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 04:37:01,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:37:01,944 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 04:37:01,945 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 04:37:01,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 04:37:02.206 UTC [5197] WARNING: there is no transaction in progress
2026-08-09 04:37:02,214 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 04:37:02,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:37:02,483 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 04:37:02,483 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 04:37:02,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 04:37:02,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 04:37:02,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-09 04:37:03,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 04:37:03,502 [I] Table x_portal_user does not exist in database ranger
2026-08-09 04:37:03,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 04:37:03,747 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 04:37:03,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 04:37:04,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 04:37:04,272 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 04:37:04,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 04:37:05.244 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.260 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.277 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.292 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.307 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.324 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.378 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.384 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.398 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.408 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.419 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.434 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.444 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.451 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.458 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:05.464 UTC [5353] WARNING: there is no transaction in progress
2026-08-09 04:37:06,079 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 04:37:06,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 04:37:06,324 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 04:37:06,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 04:37:06,602 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 04:37:06,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 04:37:06,872 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 04:37:06,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 04:37:07,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 04:37:07,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-09 04:37:07,677 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 04:37:07,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 04:37:07,915 [I] DB_PATCHES have already been applied
I20260809 04:37:07.921367 19824 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWTI4ci/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 04:37:07.921434 19824 mini_ranger.cc:193] Using host: 127.19.92.20
I20260809 04:37:07.926047 19824 mini_ranger.cc:227] Ranger admin URL: http://127.19.92.20:43671
Aug 09, 2026 4:37:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:37:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:37:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 4:37:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 4:37:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 4:37:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 4:37:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 04:37:09.102509 19824 mini_ranger.cc:160] Time spent starting Ranger: real 8.659s user 0.000s sys 0.011s
Aug 09, 2026 4:37:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 4:37:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 09, 2026 4:37:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 4:37:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 4:37:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 04:37:31.096212 19824 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.92.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.92.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/rangerkms_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.92.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/rangerkms_127.19.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.92.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.92.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.92.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/HTTP_127.19.92.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.92.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/HTTP_127.19.92.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 04:37:31 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250251, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 04:37:31.211186 19824 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 04:37:31.222977 19824 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 04:37:32.019448 19824 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 04:37:32.019524 19824 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms
I20260809 04:37:32.019541 19824 mini_ranger_kms.cc:227] Using postgres at 127.19.92.20:48199
2026-08-09 04:37:32,184 [I] DB FLAVOR :POSTGRES
2026-08-09 04:37:32,184 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 04:37:32,184 [I] Checking connection
2026-08-09 04:37:32,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 04:37:32,430 [I] connection success
2026-08-09 04:37:32,431 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 04:37:32,431 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 04:37:32,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 04:37:32,694 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 04:37:32,695 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 04:37:32,695 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 04:37:32,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.92.20:48199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 04:37:32,975 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 04:37:33.412042 19824 mini_ranger_kms.cc:310] Created kms service
I20260809 04:37:33.590481 19824 mini_ranger_kms.cc:326] Created kudu user
I20260809 04:37:33.636217 19824 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 04:37:33 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250251, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:34.065825 19824 mini_ranger_kms.cc:384] Added ranger policy
I20260809 04:37:34.066119 19824 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWTI4ci/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWTI4ci/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 04:37:34.066151 19824 mini_ranger_kms.cc:240] Using host: 127.19.92.20
I20260809 04:37:34.071393 19824 mini_ranger_kms.cc:276] Ranger KMS PID: 5627
I20260809 04:37:34.071507 19824 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.92.20:59467
Aug 09, 2026 4:37:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:37:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:37:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 4:37:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 4:37:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 4:37:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 04:37:36 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786250256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.92.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:36.559670 19824 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.348s user 0.023s sys 0.128s
I20260809 04:37:36.559821 19824 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 04:37:36.559908 19824 mini_ranger_kms.cc:401] 127.19.92.20:59467/kms/v1/keys
Aug 09 04:37:36 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.92.20@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
Aug 09 04:37:37 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.92.20@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
Aug 09 04:37:39 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:37:39.455681 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:37249
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:37249
--ranger_config_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:59467/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:39.573244 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:39.573619 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:39.573671 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:39.577863 5711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:39.577935 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:39.577952 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:39.577967 5711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:39.577981 5711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:39.583827 5711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.92.20:59467/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:37249
--ranger_config_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:39.584988 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:39.586078 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:39.595356 5721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:39.595467 5723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:39.595618 5711 server_base.cc:1034] running on GCE node
W20260809 04:37:39.595376 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:39.596321 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:39.597625 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:39.598810 5711 hybrid_clock.cc:648] HybridClock initialized: now 1786250259598788 us; error 44 us; skew 500 ppm
Aug 09 04:37:39 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250259, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:39.601697 5711 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:37:39.602727 5711 webserver.cc:460] Webserver started at http://127.19.92.62:37683/ using document root <none> and password file <none>
I20260809 04:37:39.603258 5711 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:39.603312 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:39.603606 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:37:39 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250259, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:39.668912 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "43c660122e7e44d98e8d2c8b587182c5"
format_stamp: "Formatted at 2026-08-09 04:37:39 on dist-test-slave-rmhg"
server_key: "6d5e09131049e0584483628a01efc4d3"
server_key_iv: "02ff80642c79d93bb9cb187d2df35dec"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:39.669775 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "43c660122e7e44d98e8d2c8b587182c5"
format_stamp: "Formatted at 2026-08-09 04:37:39 on dist-test-slave-rmhg"
server_key: "6d5e09131049e0584483628a01efc4d3"
server_key_iv: "02ff80642c79d93bb9cb187d2df35dec"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:39.675148 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:37:39.693192 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:39.694958 5711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:37:39.695114 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "43c660122e7e44d98e8d2c8b587182c5"
format_stamp: "Formatted at 2026-08-09 04:37:39 on dist-test-slave-rmhg"
server_key: "6d5e09131049e0584483628a01efc4d3"
server_key_iv: "02ff80642c79d93bb9cb187d2df35dec"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:39.695259 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:39.720034 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:39.723683 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:39.723898 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:39.732901 5821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:37249 every 8 connection(s)
I20260809 04:37:39.732913 5711 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:37249
I20260809 04:37:39.734230 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:37:39.735050 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 5711
I20260809 04:37:39.735179 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:37:39.735456 19824 external_mini_cluster.cc:1428] Setting key 477423393a63ca726ea948a02bc5eef9
I20260809 04:37:39.737989 5823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 04:37:39 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:39.745474 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Bootstrap starting.
I20260809 04:37:39.748401 5823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:39.749356 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:39.737001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.19.92.62:37249)
0809 04:37:39.737466 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:39.737484 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:39.737523 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:37:39.738209 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:39.738228 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:39.738582 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:39.738910 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:39.739730 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:39.740760 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:39.741670 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:39.742194 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:39.742365 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:39.744846 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:39.744852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:39.744868 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:39.744900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:39.746484 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:39.746908 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:39.746914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:39.746918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:39.747029 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:39.747297 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:39.747301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:39.747303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:39.747751 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:39.747936 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:39.748233 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 04:37:39.748554 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":159,"threads_started":1}
I20260809 04:37:39.749989 5823 log.cc:824] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:39.752444 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: No bootstrap required, opened a new log
I20260809 04:37:39.755978 5823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } }
I20260809 04:37:39.756230 5823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:39.756305 5823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c660122e7e44d98e8d2c8b587182c5, State: Initialized, Role: FOLLOWER
I20260809 04:37:39.756783 5823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } }
I20260809 04:37:39.756954 5823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:37:39.757022 5823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:37:39.757125 5823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:39.758113 5823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } }
I20260809 04:37:39.758495 5823 leader_election.cc:302] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c660122e7e44d98e8d2c8b587182c5; no voters:
I20260809 04:37:39.758908 5823 leader_election.cc:288] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:37:39.759153 5829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:37:39.759367 5829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 1 LEADER]: Becoming Leader. State: Replica: 43c660122e7e44d98e8d2c8b587182c5, State: Running, Role: LEADER
I20260809 04:37:39.759748 5829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } }
I20260809 04:37:39.760408 5823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:39.761088 5829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c660122e7e44d98e8d2c8b587182c5. Latest consensus state: current_term: 1 leader_uuid: "43c660122e7e44d98e8d2c8b587182c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } } }
I20260809 04:37:39.761240 5829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:39.761538 5833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43c660122e7e44d98e8d2c8b587182c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } } }
I20260809 04:37:39.761680 5833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:39.762792 5823 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 04:37:39.763998 5843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:37:39.770145 5843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:37:39.780130 5843 catalog_manager.cc:1347] Generated new cluster ID: 305018ee9f1e4ba3b76e70f35604dcb6
I20260809 04:37:39.780304 5843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:37:39.810987 5843 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:37:39.812477 5843 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:37:39.826916 5843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Generated new TSK 0
I20260809 04:37:39.827903 5843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:37:40.576077 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:59467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37249
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:40.724073 5894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:40.724524 5894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:40.724669 5894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:40.730867 5894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:40.730964 5894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:40.731053 5894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:37:40.739152 5894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.92.20:59467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:40.740741 5894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:40.742193 5894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:40.753978 5907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:40.753978 5906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:40.754220 5909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:40.754796 5894 server_base.cc:1034] running on GCE node
I20260809 04:37:40.755245 5894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:40.755949 5894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:40.757138 5894 hybrid_clock.cc:648] HybridClock initialized: now 1786250260757106 us; error 42 us; skew 500 ppm
Aug 09 04:37:40 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:40.760087 5894 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:37:40.761135 5894 webserver.cc:460] Webserver started at http://127.19.92.1:43391/ using document root <none> and password file <none>
I20260809 04:37:40.761864 5894 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:40.761937 5894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:40.762192 5894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:37:40 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:40.790719 5894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "f3b693d0001441d08bedb0a7d0e24737"
format_stamp: "Formatted at 2026-08-09 04:37:40 on dist-test-slave-rmhg"
server_key: "18ef9c1f7d6eaf4fbc44e95b0ce9f8a6"
server_key_iv: "a49d7482b6496423cd4e2eb9ea300ee6"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:40.791682 5894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "f3b693d0001441d08bedb0a7d0e24737"
format_stamp: "Formatted at 2026-08-09 04:37:40 on dist-test-slave-rmhg"
server_key: "18ef9c1f7d6eaf4fbc44e95b0ce9f8a6"
server_key_iv: "a49d7482b6496423cd4e2eb9ea300ee6"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:40.801770 5894 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260809 04:37:40.823251 5919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:40.825045 5894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:37:40.825203 5894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "f3b693d0001441d08bedb0a7d0e24737"
format_stamp: "Formatted at 2026-08-09 04:37:40 on dist-test-slave-rmhg"
server_key: "18ef9c1f7d6eaf4fbc44e95b0ce9f8a6"
server_key_iv: "a49d7482b6496423cd4e2eb9ea300ee6"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:40.825340 5894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:40.854518 5894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:40.858158 5894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:40.858438 5894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:40.859236 5894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:40.860409 5894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:40.860599 5894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:40.860733 5894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:40.860822 5894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:40.875032 5894 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:38615
I20260809 04:37:40.876416 5894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:37:40.880584 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 5894
I20260809 04:37:40.880720 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:37:40.881091 19824 external_mini_cluster.cc:1428] Setting key 32c5b63557448565966ec37126c3d28c
I20260809 04:37:40.877341 6089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:38615 every 8 connection(s)
Aug 09 04:37:40 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
Aug 09 04:37:40 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786250260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:40.900941 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:40.880558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37249 (local address 127.19.92.1:33695)
0809 04:37:40.882526 (+ 1968us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:40.882586 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:40.883598 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:40.884531 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:40.884541 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:40.884885 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:40.885632 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:40.885651 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:40.886830 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:40.886833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:40.888265 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:40.888277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:40.889061 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:40.889065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:40.889309 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:40.890169 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:40.890192 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:40.891964 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:40.897068 (+ 5104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:40.897076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:40.897099 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:40.897537 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:40.897909 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:40.897912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:40.897914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:40.898126 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:40.898762 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:40.898770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:40.899182 (+ 412us) client_negotiation.cc:769] Sending connection context
0809 04:37:40.899401 (+ 219us) client_negotiation.cc:240] Negotiation successful
0809 04:37:40.899800 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"thread_start_us":153,"threads_started":1}
I20260809 04:37:40.901885 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:40.879989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:33695 (local address 127.19.92.62:37249)
0809 04:37:40.884014 (+ 4025us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:40.884019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:40.884032 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:37:40.884070 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:40.884073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:40.884117 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:40.884195 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:40.885807 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:40.886675 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:40.888406 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:40.888770 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:40.888870 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:40.892167 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:40.892171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:40.892174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:40.892205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:40.896897 (+ 4692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:40.897689 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:40.897694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:40.897696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:40.897808 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:40.898254 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:40.898258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:40.898260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:40.898640 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:40.898751 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:40.901554 (+ 2803us) server_negotiation.cc:299] Negotiation successful
0809 04:37:40.901708 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3940,"thread_start_us":73,"threads_started":1}
I20260809 04:37:40.903041 6090 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37249
I20260809 04:37:40.903298 6090 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:40.903926 6090 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:40.905818 5770 ts_manager.cc:194] Registered new tserver with Master: f3b693d0001441d08bedb0a7d0e24737 (127.19.92.1:38615)
I20260809 04:37:40.907143 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:33695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:37:40.920082 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:59467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37249
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:41.117580 6106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:41.118148 6106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:41.118254 6106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:41.125028 6106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:41.125224 6106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:41.125366 6106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:37:41.133884 6106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.92.20:59467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:41.135556 6106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:41.137133 6106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:41.153026 6113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:41.153301 6106 server_base.cc:1034] running on GCE node
W20260809 04:37:41.153553 6115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:41.153600 6112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:41.154322 6106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:41.155254 6106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:41.156459 6106 hybrid_clock.cc:648] HybridClock initialized: now 1786250261156418 us; error 60 us; skew 500 ppm
Aug 09 04:37:41 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:41.160938 6106 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:37:41.162317 6106 webserver.cc:460] Webserver started at http://127.19.92.2:42735/ using document root <none> and password file <none>
I20260809 04:37:41.163101 6106 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:41.163194 6106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:41.163431 6106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:37:41 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:41.182279 6106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "be57f7efbee64597b0a065b93e29a0de"
format_stamp: "Formatted at 2026-08-09 04:37:41 on dist-test-slave-rmhg"
server_key: "d0b078aa92ad1f85bac1b5ed74230599"
server_key_iv: "244652d7fdabe645033e2602b5218edb"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:41.183120 6106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "be57f7efbee64597b0a065b93e29a0de"
format_stamp: "Formatted at 2026-08-09 04:37:41 on dist-test-slave-rmhg"
server_key: "d0b078aa92ad1f85bac1b5ed74230599"
server_key_iv: "244652d7fdabe645033e2602b5218edb"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:41.187479 6106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:37:41.209266 6126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:41.211251 6106 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260809 04:37:41.211493 6106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "be57f7efbee64597b0a065b93e29a0de"
format_stamp: "Formatted at 2026-08-09 04:37:41 on dist-test-slave-rmhg"
server_key: "d0b078aa92ad1f85bac1b5ed74230599"
server_key_iv: "244652d7fdabe645033e2602b5218edb"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:41.211753 6106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:41.247438 6106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:41.250766 6106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:41.251041 6106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:41.251814 6106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:41.252930 6106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:41.252992 6106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:41.253074 6106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:41.253101 6106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:41.271917 6106 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:38375
I20260809 04:37:41.272169 6297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:38375 every 8 connection(s)
I20260809 04:37:41.273278 6106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:37:41.282428 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 6106
I20260809 04:37:41.282578 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:37:41.282918 19824 external_mini_cluster.cc:1428] Setting key fa9a5280b88735af90eb9fc75e092fb3
Aug 09 04:37:41 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:41.290093 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:41.277109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:43503 (local address 127.19.92.62:37249)
0809 04:37:41.277278 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:41.277283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:41.277303 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:37:41.278268 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:41.278271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:41.278334 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:41.278410 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:41.279936 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:41.280502 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:41.282009 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:41.282299 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:41.282404 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:41.286254 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:41.286258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:41.286261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:41.286293 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:41.287901 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:41.288420 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:41.288423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:41.288425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:41.288507 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:41.288849 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:41.288853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:41.288854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:41.289056 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:41.289141 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:41.289750 (+ 609us) server_negotiation.cc:299] Negotiation successful
0809 04:37:41.289888 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 04:37:41.291045 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:41.275937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37249 (local address 127.19.92.2:43503)
0809 04:37:41.276622 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:41.276684 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:41.278032 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:41.278668 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:41.278676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:41.279121 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:41.279774 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:41.279792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:41.280629 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:41.280633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:41.281881 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:41.281892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:41.282596 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:41.282601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:41.282812 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:41.283530 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:41.283552 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:41.286053 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:41.288018 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:41.288025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:41.288045 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:41.288313 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:41.288589 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:41.288592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:41.288594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:41.288750 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:41.289182 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:41.289189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:41.289496 (+ 307us) client_negotiation.cc:769] Sending connection context
0809 04:37:41.289676 (+ 180us) client_negotiation.cc:240] Negotiation successful
0809 04:37:41.290026 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":154,"threads_started":1}
I20260809 04:37:41.292310 6299 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37249
I20260809 04:37:41.292569 6299 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:41.293107 6299 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:41.294196 5770 ts_manager.cc:194] Registered new tserver with Master: be57f7efbee64597b0a065b93e29a0de (127.19.92.2:38375)
I20260809 04:37:41.294765 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:43503
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:37:41.350610 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:59467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37249
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:41.571686 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:41.572225 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:41.572359 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:41.578644 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:41.578809 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:41.578938 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:37:41.587543 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.92.20:59467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:41.589304 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:41.590860 6306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 09 04:37:41 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
W20260809 04:37:41.606451 6317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:41.607729 6316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:41.608177 6306 server_base.cc:1034] running on GCE node
W20260809 04:37:41.611326 6319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:41.612048 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:41.612954 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:41.620373 6306 hybrid_clock.cc:648] HybridClock initialized: now 1786250261620261 us; error 123 us; skew 500 ppm
Aug 09 04:37:41 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:41.625771 6306 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:37:41.627384 6306 webserver.cc:460] Webserver started at http://127.19.92.3:37601/ using document root <none> and password file <none>
I20260809 04:37:41.628225 6306 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:41.628358 6306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:41.628641 6306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:37:41 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:41.656333 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "068dd133ef644f2695be67ebca3f1962"
format_stamp: "Formatted at 2026-08-09 04:37:41 on dist-test-slave-rmhg"
server_key: "1c6e709326396447ea28d22b03829a49"
server_key_iv: "6f0aedbebac9b1349b66a1c6a91393cf"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:41.657385 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "068dd133ef644f2695be67ebca3f1962"
format_stamp: "Formatted at 2026-08-09 04:37:41 on dist-test-slave-rmhg"
server_key: "1c6e709326396447ea28d22b03829a49"
server_key_iv: "6f0aedbebac9b1349b66a1c6a91393cf"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:41.671628 6306 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260809 04:37:41.693470 6328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:41.695358 6306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260809 04:37:41.695501 6306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "068dd133ef644f2695be67ebca3f1962"
format_stamp: "Formatted at 2026-08-09 04:37:41 on dist-test-slave-rmhg"
server_key: "1c6e709326396447ea28d22b03829a49"
server_key_iv: "6f0aedbebac9b1349b66a1c6a91393cf"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:41.695641 6306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:41.725674 6306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:41.729934 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:41.730244 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:41.731145 6306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:41.732435 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:41.732537 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:41.732636 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:41.732698 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:41.755647 6306 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:44979
I20260809 04:37:41.757196 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:37:41.757483 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 6306
I20260809 04:37:41.757629 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:37:41.757970 19824 external_mini_cluster.cc:1428] Setting key 36445ab90c134e6dc002f80129a8b063
I20260809 04:37:41.762346 6510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:44979 every 8 connection(s)
Aug 09 04:37:41 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:41.787868 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:41.766182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:38471 (local address 127.19.92.62:37249)
0809 04:37:41.766330 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:41.766335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:41.769883 (+ 3548us) server_negotiation.cc:407] Connection header received
0809 04:37:41.773433 (+ 3550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:41.773437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:41.773494 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:41.773546 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:41.775401 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:41.776202 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:41.777963 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:41.778296 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:41.778366 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:41.782096 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:41.782100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:41.782103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:41.782132 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:41.785279 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:41.785840 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:41.785844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:41.785846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:41.785937 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:41.786354 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:41.786357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:41.786359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:41.786590 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:41.786667 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:41.787531 (+ 864us) server_negotiation.cc:299] Negotiation successful
0809 04:37:41.787684 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:37:41.788997 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:41.764930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37249 (local address 127.19.92.3:38471)
0809 04:37:41.769667 (+ 4737us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:41.769751 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:41.773138 (+ 3387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:41.774046 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:41.774056 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:41.774502 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:41.775237 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:41.775257 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:41.776357 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:41.776361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:41.777816 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:41.777828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:41.778578 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:41.778582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:41.778785 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:41.779599 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:41.779621 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:41.781885 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:41.785408 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:41.785416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:41.785434 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:41.785733 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:41.786049 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:41.786052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:41.786054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:41.786252 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:41.786734 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:41.786741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:41.787191 (+ 450us) client_negotiation.cc:769] Sending connection context
0809 04:37:41.787409 (+ 218us) client_negotiation.cc:240] Negotiation successful
0809 04:37:41.787768 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4326,"thread_start_us":170,"threads_started":1}
I20260809 04:37:41.790373 6511 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37249
I20260809 04:37:41.790700 6511 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:41.791471 6511 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:41.792862 5770 ts_manager.cc:194] Registered new tserver with Master: 068dd133ef644f2695be67ebca3f1962 (127.19.92.3:44979)
I20260809 04:37:41.793705 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:38471
I20260809 04:37:41.795980 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:37:41.796850 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 5894
I20260809 04:37:41.908942 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 6106
I20260809 04:37:41.924314 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 6306
I20260809 04:37:41.934962 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 5711
I20260809 04:37:41.948629 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:37249
--webserver_interface=127.19.92.62
--webserver_port=37683
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:37249
--ranger_config_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:59467/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.92.20:59467/kms with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:42.062683 6518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:42.062963 6518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:42.063020 6518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:42.067072 6518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:42.067135 6518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:42.067162 6518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:42.067183 6518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:42.067212 6518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:42.072680 6518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.92.20:59467/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:37249
--ranger_config_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=37683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:42.073822 6518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:42.074862 6518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:42.083622 6527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:42.083637 6524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:42.083637 6534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:42.084022 6518 server_base.cc:1034] running on GCE node
I20260809 04:37:42.084683 6518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:42.085932 6518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:42.087112 6518 hybrid_clock.cc:648] HybridClock initialized: now 1786250262087087 us; error 37 us; skew 500 ppm
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:42.089874 6518 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:37:42.090777 6518 webserver.cc:460] Webserver started at http://127.19.92.62:37683/ using document root <none> and password file <none>
I20260809 04:37:42.091308 6518 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:42.091375 6518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:42.128531 6518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:37:42.131024 6545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:42.132588 6518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:37:42.132731 6518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "43c660122e7e44d98e8d2c8b587182c5"
format_stamp: "Formatted at 2026-08-09 04:37:39 on dist-test-slave-rmhg"
server_key: "6d5e09131049e0584483628a01efc4d3"
server_key_iv: "02ff80642c79d93bb9cb187d2df35dec"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:42.133366 6518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:42.167448 6518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:42.170274 6518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:42.170464 6518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:42.177629 6518 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:37249
I20260809 04:37:42.177649 6638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:37249 every 8 connection(s)
I20260809 04:37:42.178750 6518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:37:42.183456 6639 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:37:42.185240 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Bootstrap starting.
I20260809 04:37:42.186956 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 6518
I20260809 04:37:42.187402 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:59467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:38615
--local_ip_for_outbound_sockets=127.19.92.1
--tserver_master_addrs=127.19.92.62:37249
--webserver_port=43391
--webserver_interface=127.19.92.1
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.92.20:59467/kms with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:37:42.202646 6639 log.cc:824] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:42.211503 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:37:42.211947 6639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Bootstrap complete.
I20260809 04:37:42.217344 6639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } }
I20260809 04:37:42.217849 6639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c660122e7e44d98e8d2c8b587182c5, State: Initialized, Role: FOLLOWER
I20260809 04:37:42.218430 6639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } }
I20260809 04:37:42.218580 6639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:37:42.218648 6639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:37:42.218753 6639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:37:42.221268 6639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } }
I20260809 04:37:42.221632 6639 leader_election.cc:302] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c660122e7e44d98e8d2c8b587182c5; no voters:
I20260809 04:37:42.221999 6639 leader_election.cc:288] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 04:37:42.222107 6646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:37:42.222338 6646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [term 2 LEADER]: Becoming Leader. State: Replica: 43c660122e7e44d98e8d2c8b587182c5, State: Running, Role: LEADER
I20260809 04:37:42.222714 6646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } }
I20260809 04:37:42.223249 6639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:42.224012 6647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43c660122e7e44d98e8d2c8b587182c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } } }
I20260809 04:37:42.224131 6647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:42.224907 6649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c660122e7e44d98e8d2c8b587182c5. Latest consensus state: current_term: 2 leader_uuid: "43c660122e7e44d98e8d2c8b587182c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c660122e7e44d98e8d2c8b587182c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 37249 } } }
I20260809 04:37:42.224992 6649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:42.225302 6657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:37:42.225536 6639 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 04:37:42.231889 6657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:37:42.232882 6657 catalog_manager.cc:1259] Loaded cluster ID: 305018ee9f1e4ba3b76e70f35604dcb6
I20260809 04:37:42.232996 6657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:37:42.236670 6657 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:37:42.237740 6657 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Loaded TSK: 0
I20260809 04:37:42.239243 6657 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:37:42.317260 6642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:42.317559 6642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:42.317621 6642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:42.322083 6642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:42.322154 6642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:42.322237 6642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:37:42.328416 6642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.92.20:59467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:38615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=43391
--tserver_master_addrs=127.19.92.62:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:42.329607 6642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:42.330658 6642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:42.342576 6698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:42.342497 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:42.342983 6696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:42.343111 6642 server_base.cc:1034] running on GCE node
I20260809 04:37:42.343693 6642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:42.344467 6642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:42.345655 6642 hybrid_clock.cc:648] HybridClock initialized: now 1786250262345609 us; error 61 us; skew 500 ppm
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:42.351492 6642 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:37:42.352515 6642 webserver.cc:460] Webserver started at http://127.19.92.1:43391/ using document root <none> and password file <none>
I20260809 04:37:42.353266 6642 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:42.353372 6642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:42.386950 6642 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.002s sys 0.001s
I20260809 04:37:42.398849 6715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:42.403242 6642 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.001s
I20260809 04:37:42.403430 6642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "f3b693d0001441d08bedb0a7d0e24737"
format_stamp: "Formatted at 2026-08-09 04:37:40 on dist-test-slave-rmhg"
server_key: "18ef9c1f7d6eaf4fbc44e95b0ce9f8a6"
server_key_iv: "a49d7482b6496423cd4e2eb9ea300ee6"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:42.404294 6642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:42.421965 6642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:42.427244 6642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:42.427563 6642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:42.428473 6642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:42.429821 6642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:42.429893 6642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:42.429951 6642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:42.430039 6642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 04:37:42.443105 6642 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:38615
I20260809 04:37:42.443169 6900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:38615 every 8 connection(s)
I20260809 04:37:42.444707 6642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:37:42.452687 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 6642
I20260809 04:37:42.453229 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:59467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:38375
--local_ip_for_outbound_sockets=127.19.92.2
--tserver_master_addrs=127.19.92.62:37249
--webserver_port=42735
--webserver_interface=127.19.92.2
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.92.20:59467/kms with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:42.473457 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:42.447630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37249 (local address 127.19.92.1:54259)
0809 04:37:42.449278 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:42.449340 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:42.450467 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:42.452542 (+ 2075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:42.452551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:42.452886 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:42.453507 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:42.453525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:42.461762 (+ 8237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:42.461766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:42.463235 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:42.463250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:42.464398 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:42.464403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:42.464664 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:42.465425 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:42.465449 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:42.467281 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:42.469944 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:42.469951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:42.469968 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:42.470261 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:42.470644 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:42.470647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:42.470649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:42.470804 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:42.471458 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:42.471465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:42.471770 (+ 305us) client_negotiation.cc:769] Sending connection context
0809 04:37:42.472000 (+ 230us) client_negotiation.cc:240] Negotiation successful
0809 04:37:42.472327 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"thread_start_us":149,"threads_started":1}
I20260809 04:37:42.475600 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:42.447481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:54259 (local address 127.19.92.62:37249)
0809 04:37:42.448909 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:42.448936 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:42.450820 (+ 1884us) server_negotiation.cc:407] Connection header received
0809 04:37:42.451169 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:42.451186 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:42.451638 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:42.452025 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:42.453688 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:42.461544 (+ 7856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:42.463418 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:42.464189 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:42.465117 (+ 928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:42.467542 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:42.467549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:42.467570 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:42.467608 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:42.469787 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:42.470400 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:42.470407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:42.470414 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:42.470542 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:42.470907 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:42.470910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:42.470912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:42.471336 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:42.471467 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:42.474372 (+ 2905us) server_negotiation.cc:299] Negotiation successful
0809 04:37:42.474688 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":134,"threads_started":1}
I20260809 04:37:42.478894 6902 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37249
I20260809 04:37:42.479189 6902 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:42.479825 6902 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
W20260809 04:37:42.482638 6902 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:37249 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 04:37:42.482772 6902 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:42.482954 6902 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:42.483946 6902 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:42.484139 6902 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
W20260809 04:37:42.485023 6902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:37:42.638782 6907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:42.639249 6907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:42.639345 6907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:42.645601 6907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:42.645691 6907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:42.645784 6907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:37:42.654081 6907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.92.20:59467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:38375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=42735
--tserver_master_addrs=127.19.92.62:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:42.655687 6907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:42.657178 6907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:42.669674 6913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:42.670185 6914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:42.671720 6907 server_base.cc:1034] running on GCE node
W20260809 04:37:42.671831 6916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:42.672372 6907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:42.673499 6907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:42.674714 6907 hybrid_clock.cc:648] HybridClock initialized: now 1786250262674672 us; error 65 us; skew 500 ppm
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:42.678571 6907 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:37:42.679906 6907 webserver.cc:460] Webserver started at http://127.19.92.2:42735/ using document root <none> and password file <none>
I20260809 04:37:42.680734 6907 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:42.680841 6907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:42.701313 6907 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:37:42.704697 6928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:42.706435 6907 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:37:42.706583 6907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "be57f7efbee64597b0a065b93e29a0de"
format_stamp: "Formatted at 2026-08-09 04:37:41 on dist-test-slave-rmhg"
server_key: "d0b078aa92ad1f85bac1b5ed74230599"
server_key_iv: "244652d7fdabe645033e2602b5218edb"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:42.707315 6907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:42.726013 6907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:42.729730 6907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:42.729996 6907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:42.730937 6907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:42.732069 6907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:42.732170 6907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:42.732259 6907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:42.732313 6907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:42.742975 6907 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:38375
I20260809 04:37:42.743031 7108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:38375 every 8 connection(s)
I20260809 04:37:42.744102 6907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:37:42.750667 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 6907
I20260809 04:37:42.751209 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.92.20:59467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:44979
--local_ip_for_outbound_sockets=127.19.92.3
--tserver_master_addrs=127.19.92.62:37249
--webserver_port=37601
--webserver_interface=127.19.92.3
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.92.20:59467/kms with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:42.760126 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:42.746132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:51377 (local address 127.19.92.62:37249)
0809 04:37:42.746268 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:42.746272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:42.748054 (+ 1782us) server_negotiation.cc:407] Connection header received
0809 04:37:42.749266 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:42.749270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:42.749337 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:42.749416 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:42.750872 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:42.751641 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:42.753404 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:42.753696 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:42.753764 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:42.756461 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:42.756465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:42.756468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:42.756501 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:42.758003 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:42.758541 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:42.758544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:42.758546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:42.758630 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:42.758960 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:42.758964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:42.758966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:42.759159 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:42.759246 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:42.759793 (+ 547us) server_negotiation.cc:299] Negotiation successful
0809 04:37:42.759952 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:37:42.761047 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:42.746602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37249 (local address 127.19.92.2:51377)
0809 04:37:42.747867 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:42.747931 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:42.749013 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:42.749617 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:42.749627 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:42.750002 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:42.750672 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:42.750695 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:42.751820 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:42.751823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:42.753217 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:42.753232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:42.753966 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:42.753969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:42.754133 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:42.754766 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:42.754785 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:42.756279 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:42.758125 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:42.758132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:42.758147 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:42.758402 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:42.758716 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:42.758720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:42.758722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:42.758863 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:42.759262 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:42.759270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:42.759557 (+ 287us) client_negotiation.cc:769] Sending connection context
0809 04:37:42.759739 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 04:37:42.760078 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":150,"threads_started":1}
I20260809 04:37:42.762235 7109 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37249
I20260809 04:37:42.762493 7109 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:42.762998 7109 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
W20260809 04:37:42.764530 7109 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:37249 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 04:37:42.764652 7109 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:42.764822 7109 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:42.765679 7109 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:42.765900 7109 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
W20260809 04:37:42.766744 7109 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:37:42.899372 7113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:42.899746 7113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:42.899811 7113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:42.905944 7113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:42.906032 7113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:42.906121 7113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:37:42.914279 7113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.92.20:59467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:44979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=37601
--tserver_master_addrs=127.19.92.62:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:42.915793 7113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:42.916921 7113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:42.927384 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:42.927735 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:42.927855 7125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:42.929059 7113 server_base.cc:1034] running on GCE node
I20260809 04:37:42.929738 7113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:42.930529 7113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:42.931725 7113 hybrid_clock.cc:648] HybridClock initialized: now 1786250262931698 us; error 44 us; skew 500 ppm
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:42.935021 7113 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:37:42.935949 7113 webserver.cc:460] Webserver started at http://127.19.92.3:37601/ using document root <none> and password file <none>
I20260809 04:37:42.936501 7113 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:42.936576 7113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 04:37:42 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:42.953923 7113 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:37:42.958412 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:42.962329 7113 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260809 04:37:42.962477 7113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "068dd133ef644f2695be67ebca3f1962"
format_stamp: "Formatted at 2026-08-09 04:37:41 on dist-test-slave-rmhg"
server_key: "1c6e709326396447ea28d22b03829a49"
server_key_iv: "6f0aedbebac9b1349b66a1c6a91393cf"
server_key_version: "kuduclusterkey@0"
I20260809 04:37:42.963093 7113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:42.982113 7113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:42.986615 7113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:42.986855 7113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:42.987629 7113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:42.988662 7113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:42.988713 7113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:42.988755 7113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:42.988782 7113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:43.001142 7113 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:44979
I20260809 04:37:43.002663 7113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:37:43.003894 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:44979 every 8 connection(s)
I20260809 04:37:43.009485 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 7113
Aug 09 04:37:43 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250262, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:43.026628 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:43.010273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.19.92.62:37249)
0809 04:37:43.010542 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:43.010546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:43.015819 (+ 5273us) server_negotiation.cc:407] Connection header received
0809 04:37:43.015967 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:43.015971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:43.016019 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:43.016105 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:43.017064 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:43.017732 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:43.018811 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:43.019058 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:43.019148 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:43.020822 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:43.020826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:43.020829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:43.020859 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:43.023172 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:43.024550 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:43.024554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:43.024555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:43.024662 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:43.025395 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:43.025399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:43.025401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:43.025689 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:43.025893 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:43.026231 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 04:37:43.026426 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260809 04:37:43.029598 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:43.007806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:39605 (local address 127.19.92.62:37249)
0809 04:37:43.007976 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:43.007980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:43.014321 (+ 6341us) server_negotiation.cc:407] Connection header received
0809 04:37:43.015507 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:43.015511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:43.015572 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:43.015674 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:43.018486 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:43.019312 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:43.021460 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:43.021686 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:43.021775 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:43.024776 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:43.024780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:43.024783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:43.024811 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:43.026476 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:43.027138 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:43.027142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:43.027144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:43.027359 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:43.028006 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:43.028010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:43.028012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:43.028240 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:43.028445 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:43.029259 (+ 814us) server_negotiation.cc:299] Negotiation successful
0809 04:37:43.029436 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:37:43.031049 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:43.006551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:37249 (local address 127.19.92.3:39605)
0809 04:37:43.014064 (+ 7513us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:43.014148 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:43.015246 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:43.016278 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:43.016291 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:43.016675 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:43.018307 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:43.018330 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:43.019458 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:43.019461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:43.021344 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:43.021359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:43.021835 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:43.021840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:43.022056 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:43.022751 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:43.022773 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:43.024494 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:43.026601 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:43.026608 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:43.026630 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:43.027010 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:43.027479 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:43.027483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:43.027485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:43.027864 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:43.028519 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:43.028526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:43.028915 (+ 389us) client_negotiation.cc:769] Sending connection context
0809 04:37:43.029183 (+ 268us) client_negotiation.cc:240] Negotiation successful
0809 04:37:43.029566 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7191,"thread_start_us":173,"threads_started":1}
I20260809 04:37:43.032537 7313 heartbeater.cc:347] Connected to a master server at 127.19.92.62:37249
I20260809 04:37:43.032847 7313 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:43.033550 7313 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
W20260809 04:37:43.035432 7313 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:37249 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 04:37:43.035583 7313 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:43.035795 7313 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:43.036669 7313 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:43.036830 7313 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
W20260809 04:37:43.037600 7313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 04:37:43 dist-test-slave-rmhg krb5kdc[5027](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786250263, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:43.485301 6902 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:43.485641 6902 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:43.487030 6574 ts_manager.cc:194] Registered new tserver with Master: f3b693d0001441d08bedb0a7d0e24737 (127.19.92.1:38615)
I20260809 04:37:43.488813 6574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:54259
I20260809 04:37:43.767030 7109 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:43.767392 7109 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:43.768433 6574 ts_manager.cc:194] Registered new tserver with Master: be57f7efbee64597b0a065b93e29a0de (127.19.92.2:38375)
I20260809 04:37:43.769299 6574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:51377
Aug 09 04:37:43 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250263, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
I20260809 04:37:44.037755 7313 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:44.038098 7313 heartbeater.cc:511] Master 127.19.92.62:37249 requested a full tablet report, sending...
I20260809 04:37:44.038970 6573 ts_manager.cc:194] Registered new tserver with Master: 068dd133ef644f2695be67ebca3f1962 (127.19.92.3:44979)
I20260809 04:37:44.039767 6573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:39605
I20260809 04:37:44.051658 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:37:44.066601 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.053563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.19.92.62:37249)
0809 04:37:44.057382 (+ 3819us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:44.057386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:44.057402 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:37:44.057462 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:44.057465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:44.057517 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:44.057615 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:44.058700 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.059567 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.060677 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.061053 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.061182 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.062836 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:44.062840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:44.062843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:44.062874 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:44.064640 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:44.065140 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:44.065144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:44.065146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:44.065280 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:44.065579 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:44.065583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:44.065585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:44.065846 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:44.065988 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:44.066241 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 04:37:44.066393 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3725,"thread_start_us":110,"threads_started":1}
I20260809 04:37:44.073104 6573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:37:44.077948 6573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:37:44.103227 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.094410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38615 (local address 127.0.0.1:58774)
0809 04:37:44.096130 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:44.096144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:44.096242 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:44.096911 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:44.096914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:44.096932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:44.097013 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.097018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.099983 (+ 2965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.099986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.101357 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.101369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.102663 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.102666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.102773 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.102812 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 04:37:44.102919 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 04:37:44.103046 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1654,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260809 04:37:44.104137 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.094568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.19.92.1:38615)
0809 04:37:44.095462 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:44.095468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:44.096322 (+ 854us) server_negotiation.cc:407] Connection header received
0809 04:37:44.096400 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:44.096405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:44.096594 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:44.096733 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:44.098654 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.099858 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.101512 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.102335 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.102435 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.102599 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:44.103860 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0809 04:37:44.103974 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":84,"threads_started":1}
I20260809 04:37:44.106786 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.093137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:38375 (local address 127.0.0.1:39238)
0809 04:37:44.094123 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:44.094175 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:44.095175 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:44.096025 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:44.096031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:44.096056 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:44.096222 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.096229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.100223 (+ 3994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.100227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.102477 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.102490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.106407 (+ 3917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.106411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.106504 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.106529 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:37:44.106574 (+ 45us) client_negotiation.cc:240] Negotiation successful
0809 04:37:44.106646 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":1108,"thread_start_us":123,"threads_started":1}
I20260809 04:37:44.107688 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.093315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.19.92.2:38375)
0809 04:37:44.093813 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:44.093820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:44.095535 (+ 1715us) server_negotiation.cc:407] Connection header received
0809 04:37:44.095622 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:44.095628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:44.095850 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:44.097213 (+ 1363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:44.097386 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.098518 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.103503 (+ 4985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.104389 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.107240 (+ 2851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.107404 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:44.107456 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 04:37:44.107554 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":148,"threads_started":1}
I20260809 04:37:44.107311 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.094877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.19.92.3:44979)
0809 04:37:44.095962 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:44.095968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:44.096526 (+ 558us) server_negotiation.cc:407] Connection header received
0809 04:37:44.096746 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:44.096752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:44.096938 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:44.097058 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:44.097517 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.099464 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.105580 (+ 6116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.106451 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.106568 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.106729 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:44.107065 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 04:37:44.107174 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260809 04:37:44.108664 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.094728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:44979 (local address 127.0.0.1:44378)
0809 04:37:44.096421 (+ 1693us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:44.096446 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:44.096600 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:44.097261 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:44.097264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:44.097283 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:44.097370 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.097377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.099611 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.099614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.105396 (+ 5782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.105409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.106796 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.106800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.106899 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.106922 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:37:44.108444 (+ 1522us) client_negotiation.cc:240] Negotiation successful
0809 04:37:44.108529 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"mutex_wait_us":1337,"thread_start_us":96,"threads_started":1}
I20260809 04:37:44.109014 6768 tablet_service.cc:1467] Processing CreateTablet for tablet cd13b010ce0a486a89da067761896d5e (DEFAULT_TABLE table=test-table [id=e6ece31b0ae441829f4f58a93ad9624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:44.110401 6768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd13b010ce0a486a89da067761896d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:44.110787 7010 tablet_service.cc:1467] Processing CreateTablet for tablet cd13b010ce0a486a89da067761896d5e (DEFAULT_TABLE table=test-table [id=e6ece31b0ae441829f4f58a93ad9624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:44.111864 7010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd13b010ce0a486a89da067761896d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:44.111584 7186 tablet_service.cc:1467] Processing CreateTablet for tablet cd13b010ce0a486a89da067761896d5e (DEFAULT_TABLE table=test-table [id=e6ece31b0ae441829f4f58a93ad9624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:44.112573 7186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd13b010ce0a486a89da067761896d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:44.122671 7360 tablet_bootstrap.cc:492] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737: Bootstrap starting.
I20260809 04:37:44.123136 7362 tablet_bootstrap.cc:492] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Bootstrap starting.
I20260809 04:37:44.123279 7366 tablet_bootstrap.cc:492] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Bootstrap starting.
I20260809 04:37:44.125748 7360 tablet_bootstrap.cc:654] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:44.125748 7366 tablet_bootstrap.cc:654] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:44.126679 7366 log.cc:824] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:44.128156 7362 tablet_bootstrap.cc:654] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:44.129071 7362 log.cc:824] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:44.129494 7366 tablet_bootstrap.cc:492] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: No bootstrap required, opened a new log
I20260809 04:37:44.129715 7366 ts_tablet_manager.cc:1397] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 04:37:44.130188 7360 log.cc:824] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:44.133219 7362 tablet_bootstrap.cc:492] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: No bootstrap required, opened a new log
I20260809 04:37:44.133527 7362 ts_tablet_manager.cc:1397] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260809 04:37:44.134557 7366 raft_consensus.cc:348] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.134865 7366 raft_consensus.cc:374] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:44.134922 7366 raft_consensus.cc:729] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068dd133ef644f2695be67ebca3f1962, State: Initialized, Role: FOLLOWER
I20260809 04:37:44.135465 7366 consensus_queue.cc:260] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.136659 7366 ts_tablet_manager.cc:1428] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 04:37:44.137069 7313 heartbeater.cc:503] Master 127.19.92.62:37249 was elected leader, sending a full tablet report...
I20260809 04:37:44.140270 7360 tablet_bootstrap.cc:492] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737: No bootstrap required, opened a new log
I20260809 04:37:44.140507 7360 ts_tablet_manager.cc:1397] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20260809 04:37:44.138393 7362 raft_consensus.cc:348] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.140743 7362 raft_consensus.cc:374] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:44.140846 7362 raft_consensus.cc:729] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be57f7efbee64597b0a065b93e29a0de, State: Initialized, Role: FOLLOWER
I20260809 04:37:44.141479 7362 consensus_queue.cc:260] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.142699 7362 ts_tablet_manager.cc:1428] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260809 04:37:44.143126 7109 heartbeater.cc:503] Master 127.19.92.62:37249 was elected leader, sending a full tablet report...
I20260809 04:37:44.144585 7360 raft_consensus.cc:348] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.144834 7360 raft_consensus.cc:374] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:44.144889 7360 raft_consensus.cc:729] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b693d0001441d08bedb0a7d0e24737, State: Initialized, Role: FOLLOWER
I20260809 04:37:44.145409 7360 consensus_queue.cc:260] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.146678 7360 ts_tablet_manager.cc:1428] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:37:44.147047 6902 heartbeater.cc:503] Master 127.19.92.62:37249 was elected leader, sending a full tablet report...
Aug 09 04:37:44 dist-test-slave-rmhg krb5kdc[5027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250263, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for HTTP/127.19.92.20@KRBTEST.COM
W20260809 04:37:44.197664 6903 tablet.cc:2369] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:37:44.246713 7110 tablet.cc:2369] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:37:44.258671 7316 tablet.cc:2369] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:37:44.388540 7372 raft_consensus.cc:482] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:37:44.388804 7372 raft_consensus.cc:504] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.390388 7372 leader_election.cc:288] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be57f7efbee64597b0a065b93e29a0de (127.19.92.2:38375), f3b693d0001441d08bedb0a7d0e24737 (127.19.92.1:38615)
I20260809 04:37:44.396595 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.391295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:54087 (local address 127.19.92.1:38615)
0809 04:37:44.391419 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:44.391424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:44.391964 (+ 540us) server_negotiation.cc:407] Connection header received
0809 04:37:44.392183 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:44.392187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:44.392240 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:44.392285 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:44.392797 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.393642 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.395239 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.395855 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.395934 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.396011 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:44.396350 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 04:37:44.396438 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 04:37:44.396893 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.391460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:59089 (local address 127.19.92.2:38375)
0809 04:37:44.391563 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:44.391567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:44.391965 (+ 398us) server_negotiation.cc:407] Connection header received
0809 04:37:44.392690 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:44.392694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:44.392748 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:44.392793 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:44.393296 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.394041 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.395570 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.396188 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.396275 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.396342 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:44.396677 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 04:37:44.396765 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 04:37:44.397446 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.390733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38615 (local address 127.19.92.3:54087)
0809 04:37:44.391746 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:44.391760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:44.392060 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:44.392423 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:44.392426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:44.392441 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:44.392661 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.392668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.393793 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.393796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.395095 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.395106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.396104 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.396108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.396215 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.396239 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:37:44.397111 (+ 872us) client_negotiation.cc:240] Negotiation successful
0809 04:37:44.397233 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":3,"spinlock_wait_cycles":69504,"thread_start_us":166,"threads_started":1}
I20260809 04:37:44.397460 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.390737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:38375 (local address 127.19.92.3:59089)
0809 04:37:44.391854 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:44.391870 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:44.392536 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:44.392940 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:44.392943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:44.392963 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:44.393102 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.393109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.394215 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.394218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.395410 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:44.395423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.396437 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.396441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:44.396546 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.396567 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:37:44.397128 (+ 561us) client_negotiation.cc:240] Negotiation successful
0809 04:37:44.397253 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":15,"thread_start_us":141,"threads_started":1}
I20260809 04:37:44.398337 6826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd13b010ce0a486a89da067761896d5e" candidate_uuid: "068dd133ef644f2695be67ebca3f1962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b693d0001441d08bedb0a7d0e24737" is_pre_election: true
I20260809 04:37:44.398686 6826 raft_consensus.cc:2454] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 068dd133ef644f2695be67ebca3f1962 in term 0.
I20260809 04:37:44.399202 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd13b010ce0a486a89da067761896d5e" candidate_uuid: "068dd133ef644f2695be67ebca3f1962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be57f7efbee64597b0a065b93e29a0de" is_pre_election: true
I20260809 04:37:44.399480 7013 raft_consensus.cc:2454] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 068dd133ef644f2695be67ebca3f1962 in term 0.
I20260809 04:37:44.400206 7144 leader_election.cc:302] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068dd133ef644f2695be67ebca3f1962, be57f7efbee64597b0a065b93e29a0de; no voters:
I20260809 04:37:44.400527 7372 raft_consensus.cc:2781] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:37:44.400610 7372 raft_consensus.cc:482] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:37:44.400646 7372 raft_consensus.cc:3037] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:44.402484 7372 raft_consensus.cc:504] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.402859 7372 leader_election.cc:288] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [CANDIDATE]: Term 1 election: Requested vote from peers be57f7efbee64597b0a065b93e29a0de (127.19.92.2:38375), f3b693d0001441d08bedb0a7d0e24737 (127.19.92.1:38615)
I20260809 04:37:44.403645 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd13b010ce0a486a89da067761896d5e" candidate_uuid: "068dd133ef644f2695be67ebca3f1962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be57f7efbee64597b0a065b93e29a0de"
I20260809 04:37:44.403772 7013 raft_consensus.cc:3037] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:44.405037 7013 raft_consensus.cc:2454] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 068dd133ef644f2695be67ebca3f1962 in term 1.
I20260809 04:37:44.405427 7144 leader_election.cc:302] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068dd133ef644f2695be67ebca3f1962, be57f7efbee64597b0a065b93e29a0de; no voters:
I20260809 04:37:44.405725 6826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd13b010ce0a486a89da067761896d5e" candidate_uuid: "068dd133ef644f2695be67ebca3f1962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b693d0001441d08bedb0a7d0e24737"
I20260809 04:37:44.405853 6826 raft_consensus.cc:3037] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:44.407063 6826 raft_consensus.cc:2454] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 068dd133ef644f2695be67ebca3f1962 in term 1.
I20260809 04:37:44.407555 7372 raft_consensus.cc:2781] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:37:44.407801 7372 raft_consensus.cc:686] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 1 LEADER]: Becoming Leader. State: Replica: 068dd133ef644f2695be67ebca3f1962, State: Running, Role: LEADER
I20260809 04:37:44.408177 7372 consensus_queue.cc:237] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } }
I20260809 04:37:44.411921 6574 catalog_manager.cc:5495] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 reported cstate change: term changed from 0 to 1, leader changed from <none> to 068dd133ef644f2695be67ebca3f1962 (127.19.92.3). New cstate: current_term: 1 leader_uuid: "068dd133ef644f2695be67ebca3f1962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "068dd133ef644f2695be67ebca3f1962" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44979 } health_report { overall_health: HEALTHY } } }
I20260809 04:37:44.467110 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:44.462772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.19.92.3:44979)
0809 04:37:44.462894 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:44.462898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:44.462912 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:37:44.463113 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:44.463116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:44.463203 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:44.463318 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:37:44.463698 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.464217 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.465425 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:44.465662 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:44.465751 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:44.466087 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:37:44.466623 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:37:44.466724 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:44.466839 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 04:37:44.466929 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 04:37:44.473672 7013 raft_consensus.cc:1261] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 1 FOLLOWER]: Refusing update from remote peer 068dd133ef644f2695be67ebca3f1962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:37:44.473709 6826 raft_consensus.cc:1261] T cd13b010ce0a486a89da067761896d5e P f3b693d0001441d08bedb0a7d0e24737 [term 1 FOLLOWER]: Refusing update from remote peer 068dd133ef644f2695be67ebca3f1962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:37:44.474504 7372 consensus_queue.cc:1035] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b693d0001441d08bedb0a7d0e24737" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 38615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:37:44.475014 7391 consensus_queue.cc:1035] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be57f7efbee64597b0a065b93e29a0de" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 38375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:37:44.488063 7407 mvcc.cc:205] Tried to move back new op lower bound from 7316481083275497472 to 7316481083019882496. Current Snapshot: MvccSnapshot[applied={T|T < 7316481083275497472}]
I20260809 04:37:44.493726 6573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:37:44.493903 6573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:37:44.496423 6573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43c660122e7e44d98e8d2c8b587182c5: Sending DeleteTablet for 3 replicas of tablet cd13b010ce0a486a89da067761896d5e
I20260809 04:37:44.497416 6768 tablet_service.cc:1514] Processing DeleteTablet for tablet cd13b010ce0a486a89da067761896d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:44 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:58774
I20260809 04:37:44.497430 7010 tablet_service.cc:1514] Processing DeleteTablet for tablet cd13b010ce0a486a89da067761896d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:44 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:39238
I20260809 04:37:44.498086 7186 tablet_service.cc:1514] Processing DeleteTablet for tablet cd13b010ce0a486a89da067761896d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:44 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:44378
I20260809 04:37:44.498346 7417 tablet_replica.cc:330] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: stopping tablet replica
I20260809 04:37:44.498657 7417 raft_consensus.cc:2229] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:37:44.498967 7417 raft_consensus.cc:2258] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:37:44.499043 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 6642
I20260809 04:37:44.507469 7417 ts_tablet_manager.cc:1905] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:37:44.509354 7421 tablet_replica.cc:330] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: stopping tablet replica
I20260809 04:37:44.509903 7421 raft_consensus.cc:2229] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:37:44.510296 7417 ts_tablet_manager.cc:1918] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:37:44.510399 7417 log.cc:1196] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/cd13b010ce0a486a89da067761896d5e
I20260809 04:37:44.510535 7421 raft_consensus.cc:2258] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:37:44.510732 7417 ts_tablet_manager.cc:1939] T cd13b010ce0a486a89da067761896d5e P be57f7efbee64597b0a065b93e29a0de: Deleting consensus metadata
I20260809 04:37:44.512095 6554 catalog_manager.cc:4841] TS be57f7efbee64597b0a065b93e29a0de (127.19.92.2:38375): tablet cd13b010ce0a486a89da067761896d5e (table test-table [id=e6ece31b0ae441829f4f58a93ad9624c]) successfully deleted
I20260809 04:37:44.512300 7421 ts_tablet_manager.cc:1905] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:37:44.515193 7421 ts_tablet_manager.cc:1918] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:37:44.515384 7421 log.cc:1196] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/cd13b010ce0a486a89da067761896d5e
I20260809 04:37:44.515758 7421 ts_tablet_manager.cc:1939] T cd13b010ce0a486a89da067761896d5e P 068dd133ef644f2695be67ebca3f1962: Deleting consensus metadata
W20260809 04:37:44.516960 6546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:38615 (error 108)
I20260809 04:37:44.517553 6549 catalog_manager.cc:4841] TS 068dd133ef644f2695be67ebca3f1962 (127.19.92.3:44979): tablet cd13b010ce0a486a89da067761896d5e (table test-table [id=e6ece31b0ae441829f4f58a93ad9624c]) successfully deleted
W20260809 04:37:44.518190 7358 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:37:44.517885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:38615 (local address 127.0.0.1:58788)
0809 04:37:44.518018 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:44.518097 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:38615: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 04:37:44.518316 6546 catalog_manager.cc:4584] TS f3b693d0001441d08bedb0a7d0e24737 (127.19.92.1:38615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd13b010ce0a486a89da067761896d5e: Network error: Client connection negotiation failed: client connection to 127.19.92.1:38615: connect: Connection reset by peer (error 104)
I20260809 04:37:44.518668 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 6907
I20260809 04:37:44.543116 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 7113
I20260809 04:37:44.563009 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 6518
I20260809 04:37:44.580729 19824 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 04:37:44.926219 19824 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 04:37:44.926344 19824 mini_ranger.cc:66] Stopping Ranger...
I20260809 04:37:45.271209 19824 mini_ranger.cc:68] Stopped Ranger
2026-08-09 04:37:45.271 UTC [5081] LOG: received smart shutdown request
2026-08-09 04:37:45.276 UTC [5081] LOG: background worker "logical replication launcher" (PID 5093) exited with exit code 1
2026-08-09 04:37:45.276 UTC [5087] LOG: shutting down
2026-08-09 04:37:45.432 UTC [5081] LOG: database system is shut down
2026-08-09T04:37:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49801 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:37:45 dist-test-slave-rmhg krb5kdc[7435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:37:45 dist-test-slave-rmhg krb5kdc[7435](info): set up 2 sockets
Aug 09 04:37:45 dist-test-slave-rmhg krb5kdc[7435](info): commencing operation
krb5kdc: starting...
W20260809 04:37:46.667644 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:37:46 dist-test-slave-rmhg krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:37:46.799991 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:45579
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:45579,127.19.92.61:39459,127.19.92.60:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:46.915848 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:46.916167 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:46.916227 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:46.920367 7459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:46.920449 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:46.920475 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:46.920502 7459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:46.920526 7459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:46.926043 7459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:45579,127.19.92.61:39459,127.19.92.60:41653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:45579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:46.927188 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:46.928249 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:46.936816 7467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:46.937024 7459 server_base.cc:1034] running on GCE node
W20260809 04:37:46.936936 7469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:46.937044 7465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:46.937695 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:46.938846 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:46.940011 7459 hybrid_clock.cc:648] HybridClock initialized: now 1786250266939991 us; error 31 us; skew 500 ppm
Aug 09 04:37:46 dist-test-slave-rmhg krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250266, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:46.942776 7459 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:37:46.943715 7459 webserver.cc:460] Webserver started at http://127.19.92.62:32939/ using document root <none> and password file <none>
I20260809 04:37:46.944263 7459 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:46.944326 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:46.944546 7459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:37:46.946676 7459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb"
format_stamp: "Formatted at 2026-08-09 04:37:46 on dist-test-slave-rmhg"
server_key: "5668d97d71d498034524cb0c3f1122a7"
server_key_iv: "462f7a1954654b8817c087c8771efeb9"
server_key_version: "encryptionkey@0"
I20260809 04:37:46.947229 7459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb"
format_stamp: "Formatted at 2026-08-09 04:37:46 on dist-test-slave-rmhg"
server_key: "5668d97d71d498034524cb0c3f1122a7"
server_key_iv: "462f7a1954654b8817c087c8771efeb9"
server_key_version: "encryptionkey@0"
I20260809 04:37:46.951041 7459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:37:46.953565 7479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:46.954923 7459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:37:46.955044 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb"
format_stamp: "Formatted at 2026-08-09 04:37:46 on dist-test-slave-rmhg"
server_key: "5668d97d71d498034524cb0c3f1122a7"
server_key_iv: "462f7a1954654b8817c087c8771efeb9"
server_key_version: "encryptionkey@0"
I20260809 04:37:46.955170 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:46.968201 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:46.971742 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:46.971930 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:46.979200 7459 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:45579
I20260809 04:37:46.979210 7570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:45579 every 8 connection(s)
I20260809 04:37:46.980245 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:37:46.983170 7572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:46.987162 7572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:37:46.987947 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 7459
I20260809 04:37:46.988078 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:37:46.988379 19824 external_mini_cluster.cc:1428] Setting key 7c42f3575bfeb2296f0ee126153b088d
WARNING: no policy specified for kudu/127.19.92.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.61@KRBTEST.COM" created.
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250266, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:47.007812 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:46.989930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45579 (local address 127.0.0.1:60952)
0809 04:37:46.990729 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:46.990790 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:46.991870 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:46.992623 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:46.992629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:46.993098 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:46.993516 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:46.993544 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:46.994738 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:46.994743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:46.996022 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:46.996035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:46.996799 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:46.996804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:46.996959 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:46.997605 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:46.997630 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.002286 (+ 4656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.004397 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.004422 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.004443 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.004787 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.005224 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.005227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.005229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.005357 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.006142 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.006148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.006297 (+ 149us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.006488 (+ 191us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.006877 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":59,"thread_start_us":155,"threads_started":1}
I20260809 04:37:47.007817 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:46.989930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.19.92.62:45579)
0809 04:37:46.990551 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:46.990570 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:46.990905 (+ 335us) server_negotiation.cc:407] Connection header received
0809 04:37:46.992216 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:46.992221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:46.992410 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:46.992571 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:46.993692 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:46.994596 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:46.996169 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:46.996610 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:46.996742 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.002478 (+ 5736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.002485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.002501 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.002530 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.004257 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.004930 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.004938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.004940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.005096 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.005455 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.005458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.005460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.006025 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.006162 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.006590 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.006971 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":469,"spinlock_wait_cycles":51456,"thread_start_us":155,"threads_started":1}
I20260809 04:37:47.010715 7572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:37:47.011479 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.011204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51894)
0809 04:37:47.011330 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.011405 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 04:37:47.011677 7483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
I20260809 04:37:47.012527 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.012314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51908)
0809 04:37:47.012422 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.012470 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:37:47.012689 7572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:39459: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab.
Entry for principal kudu/127.19.92.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab.
I20260809 04:37:47.019708 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.61:39459
--webserver_interface=127.19.92.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab
--principal=kudu/127.19.92.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:45579,127.19.92.61:39459,127.19.92.60:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:37:47.061945 7572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } attempt: 1
I20260809 04:37:47.062940 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.062543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51910)
0809 04:37:47.062729 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.062816 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 04:37:47.063738 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.063492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51916)
0809 04:37:47.063604 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.063661 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 04:37:47.063911 7572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:39459: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
I20260809 04:37:47.132098 7572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } attempt: 2
I20260809 04:37:47.133100 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.132753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51924)
0809 04:37:47.132940 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.133010 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 04:37:47.133833 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.133633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51926)
0809 04:37:47.133728 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.133774 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 04:37:47.133965 7572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:39459: Network error: Client connection negotiation failed: client connection to 127.19.92.61:39459: connect: Connection refused (error 111)
W20260809 04:37:47.135049 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:47.135298 7584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:47.135367 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:47.139663 7584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:47.139734 7584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:47.139761 7584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:47.139793 7584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:47.139823 7584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:47.145457 7584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:45579,127.19.92.61:39459,127.19.92.60:41653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab
--principal=kudu/127.19.92.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.61:39459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.92.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:47.146615 7584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:47.147682 7584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:47.156421 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:47.156421 7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:47.156447 7596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:47.156905 7584 server_base.cc:1034] running on GCE node
I20260809 04:37:47.157624 7584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:47.158833 7584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:47.160050 7584 hybrid_clock.cc:648] HybridClock initialized: now 1786250267160027 us; error 37 us; skew 500 ppm
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:47.163070 7584 init.cc:377] Logged in from keytab as kudu/127.19.92.61@KRBTEST.COM (short username kudu)
I20260809 04:37:47.164037 7584 webserver.cc:460] Webserver started at http://127.19.92.61:41611/ using document root <none> and password file <none>
I20260809 04:37:47.164628 7584 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:47.164700 7584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:47.164913 7584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:37:47.167114 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/data/instance:
uuid: "10fe7a9e7079469e8c545522df7e49c4"
format_stamp: "Formatted at 2026-08-09 04:37:47 on dist-test-slave-rmhg"
server_key: "6345b1c9c0fce62d76db4052de3d21ec"
server_key_iv: "9ef6e2a070db40e59ab97354c67ab7c7"
server_key_version: "encryptionkey@0"
I20260809 04:37:47.167685 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/wal/instance:
uuid: "10fe7a9e7079469e8c545522df7e49c4"
format_stamp: "Formatted at 2026-08-09 04:37:47 on dist-test-slave-rmhg"
server_key: "6345b1c9c0fce62d76db4052de3d21ec"
server_key_iv: "9ef6e2a070db40e59ab97354c67ab7c7"
server_key_version: "encryptionkey@0"
I20260809 04:37:47.171468 7584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:37:47.173975 7616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:47.175235 7584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:37:47.175354 7584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/wal
uuid: "10fe7a9e7079469e8c545522df7e49c4"
format_stamp: "Formatted at 2026-08-09 04:37:47 on dist-test-slave-rmhg"
server_key: "6345b1c9c0fce62d76db4052de3d21ec"
server_key_iv: "9ef6e2a070db40e59ab97354c67ab7c7"
server_key_version: "encryptionkey@0"
I20260809 04:37:47.175453 7584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:47.196588 7584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:47.199744 7584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:47.199952 7584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:47.207080 7584 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.61:39459
I20260809 04:37:47.207096 7693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.61:39459 every 8 connection(s)
I20260809 04:37:47.208151 7584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
I20260809 04:37:47.211054 7696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:47.214954 7696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:37:47.217487 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 7584
I20260809 04:37:47.217602 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-1/wal/instance
I20260809 04:37:47.217902 19824 external_mini_cluster.cc:1428] Setting key 496f9be3ead6cc075cf16a78f4170bc6
I20260809 04:37:47.225131 7572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } attempt: 3
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.60@KRBTEST.COM; defaulting to no policy
I20260809 04:37:47.229849 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.216860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.19.92.62:45579)
0809 04:37:47.217017 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.217022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.218061 (+ 1039us) server_negotiation.cc:407] Connection header received
0809 04:37:47.219232 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.219235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.219286 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.219365 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.220762 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.221606 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.222745 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.223134 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.223252 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.226210 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.226213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.226215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.226241 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.227718 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.228221 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.228224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.228225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.228303 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.228636 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.228640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.228641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.228856 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.228950 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.229549 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.229679 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.60@KRBTEST.COM" created.
I20260809 04:37:47.231101 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.217318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45579 (local address 127.0.0.1:60954)
0809 04:37:47.217868 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.217930 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:47.218961 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:47.219573 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:47.219581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:47.220048 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:47.220580 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.220593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.221720 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.221723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.222610 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.222620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.223286 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.223289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.223476 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.224085 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:47.224108 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.226027 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.227829 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.227836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.227853 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.228124 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.228389 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.228392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.228394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.228545 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.228960 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.228968 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.229290 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.229470 (+ 180us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.229812 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":153,"threads_started":1}
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250266, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
I20260809 04:37:47.232934 7696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:37:47.236840 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.225671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51930)
0809 04:37:47.225880 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.225895 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:47.225973 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:47.226726 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:47.226730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:47.227207 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:47.227360 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.227369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.228527 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.228530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.229118 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.229126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.229720 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.229724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.229833 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.230256 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:47.230270 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.233022 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.235156 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.235159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.235162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.235367 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.235661 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.235665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.235667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.235754 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.236255 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.236258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.236380 (+ 122us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.236505 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.236661 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260809 04:37:47.236961 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.225811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.19.92.61:39459)
0809 04:37:47.226230 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.226242 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.226270 (+ 28us) server_negotiation.cc:407] Connection header received
0809 04:37:47.226314 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.226320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.226561 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.226936 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.227495 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.228405 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.229238 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.229607 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.229701 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.233201 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.233211 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.233225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.233251 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.235054 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.235473 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.235477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.235478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.235568 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.235836 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.235839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.235841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.236112 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.236283 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.236618 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.236770 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":98,"threads_started":1}
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
I20260809 04:37:47.240056 7572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:37:47.240887 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.240601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56024)
0809 04:37:47.240742 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.240799 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 04:37:47.241647 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.241383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56038)
0809 04:37:47.241542 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.241592 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260809 04:37:47.241849 7572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:41653: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
I20260809 04:37:47.242501 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.233792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51936)
0809 04:37:47.233930 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.233945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:47.234034 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:47.234786 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:47.234789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:47.235047 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:47.235175 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.235182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.236206 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.236210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.236922 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.236932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.237525 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.237528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.237619 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.238035 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:47.238047 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.239269 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.241030 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.241032 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.241035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.241276 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.241540 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.241543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.241544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.241625 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.242041 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.242044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.242128 (+ 84us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.242210 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.242341 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 04:37:47.242556 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.233944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.19.92.61:39459)
0809 04:37:47.234548 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.234557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.234571 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:37:47.234605 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.234608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.234650 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.234743 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.235289 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.236095 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.237057 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.237414 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.237513 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.239418 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.239421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.239423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.239443 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.240923 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.241366 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.241367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.241368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.241429 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.241715 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.241718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.241720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.241946 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.242034 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.242247 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.242412 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":535,"thread_start_us":387,"threads_started":1}
I20260809 04:37:47.243180 7696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:37:47.243745 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.243534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56040)
0809 04:37:47.243625 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.243696 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 04:37:47.243968 7618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
I20260809 04:37:47.244774 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.244557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56048)
0809 04:37:47.244658 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.244698 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:37:47.244895 7696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:41653: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab.
Entry for principal kudu/127.19.92.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab.
I20260809 04:37:47.247933 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.60:41653
--webserver_interface=127.19.92.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab
--principal=kudu/127.19.92.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:45579,127.19.92.61:39459,127.19.92.60:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:37:47.272980 7572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } attempt: 1
I20260809 04:37:47.274024 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.273646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56054)
0809 04:37:47.273827 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.273909 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 04:37:47.275033 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.274738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56070)
0809 04:37:47.274873 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.274944 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 04:37:47.275303 7572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:41653: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
I20260809 04:37:47.294162 7696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } attempt: 1
I20260809 04:37:47.295091 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.294756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56076)
0809 04:37:47.294922 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.294995 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 04:37:47.295862 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.295674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56090)
0809 04:37:47.295758 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.295805 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260809 04:37:47.295969 7696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:41653: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
I20260809 04:37:47.350488 7572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } attempt: 2
I20260809 04:37:47.351315 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.351006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56096)
0809 04:37:47.351159 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.351224 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 04:37:47.352101 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.351893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56104)
0809 04:37:47.351997 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.352045 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 04:37:47.352221 7572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:41653: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
I20260809 04:37:47.364156 7696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } attempt: 2
I20260809 04:37:47.365012 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.364712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56108)
0809 04:37:47.364857 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.364917 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 04:37:47.365777 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.365571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56122)
0809 04:37:47.365676 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.365723 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 04:37:47.365895 7696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:41653: Network error: Client connection negotiation failed: client connection to 127.19.92.60:41653: connect: Connection refused (error 111)
W20260809 04:37:47.376432 7709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:47.376688 7709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:47.376727 7709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:47.380808 7709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:47.380868 7709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:47.380887 7709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:47.380901 7709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:47.380914 7709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:47.386176 7709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:45579,127.19.92.61:39459,127.19.92.60:41653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab
--principal=kudu/127.19.92.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.60:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.92.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:47.387244 7709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:47.388353 7709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:47.396704 7719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:47.396742 7720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:47.396749 7709 server_base.cc:1034] running on GCE node
W20260809 04:37:47.396704 7726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:47.397563 7709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:47.398726 7709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:47.399885 7709 hybrid_clock.cc:648] HybridClock initialized: now 1786250267399859 us; error 36 us; skew 500 ppm
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:47.406317 7709 init.cc:377] Logged in from keytab as kudu/127.19.92.60@KRBTEST.COM (short username kudu)
I20260809 04:37:47.407289 7709 webserver.cc:460] Webserver started at http://127.19.92.60:38441/ using document root <none> and password file <none>
I20260809 04:37:47.407886 7709 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:47.407953 7709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:47.408174 7709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:37:47.410326 7709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/data/instance:
uuid: "44e92d70271c48769a4d05aff4e8585f"
format_stamp: "Formatted at 2026-08-09 04:37:47 on dist-test-slave-rmhg"
server_key: "cc3cdf7863548a489799f31e06be3c57"
server_key_iv: "3cd34407b4073276c04d230cee600b5a"
server_key_version: "encryptionkey@0"
I20260809 04:37:47.410881 7709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/wal/instance:
uuid: "44e92d70271c48769a4d05aff4e8585f"
format_stamp: "Formatted at 2026-08-09 04:37:47 on dist-test-slave-rmhg"
server_key: "cc3cdf7863548a489799f31e06be3c57"
server_key_iv: "3cd34407b4073276c04d230cee600b5a"
server_key_version: "encryptionkey@0"
I20260809 04:37:47.414629 7709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:37:47.417057 7737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:47.418293 7709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:37:47.418406 7709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/wal
uuid: "44e92d70271c48769a4d05aff4e8585f"
format_stamp: "Formatted at 2026-08-09 04:37:47 on dist-test-slave-rmhg"
server_key: "cc3cdf7863548a489799f31e06be3c57"
server_key_iv: "3cd34407b4073276c04d230cee600b5a"
server_key_version: "encryptionkey@0"
I20260809 04:37:47.418519 7709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:47.438932 7709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:47.442178 7709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:47.442350 7709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:47.449432 7709 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.60:41653
I20260809 04:37:47.449494 7824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.60:41653 every 8 connection(s)
I20260809 04:37:47.450436 7709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
I20260809 04:37:47.451375 7572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } attempt: 3
I20260809 04:37:47.453733 7825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:47.455610 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 7709
I20260809 04:37:47.455730 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250097044341-19824-0/minicluster-data/master-2/wal/instance
I20260809 04:37:47.455996 19824 external_mini_cluster.cc:1428] Setting key e616f552497ea062bdb3d9342c94167d
I20260809 04:37:47.457070 7696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } attempt: 3
I20260809 04:37:47.458755 7825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250266, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
I20260809 04:37:47.466246 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.451903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56126)
0809 04:37:47.452062 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.452080 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:47.452185 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:47.454854 (+ 2669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:47.454857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:47.455040 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:47.455151 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.455157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.456983 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.456987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.457804 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.457819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.459085 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.459090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.459190 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.459749 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:47.459764 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.461097 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.464168 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.464171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.464173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.464440 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.464826 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.464829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.464831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.464931 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.465719 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.465722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.465834 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.465921 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.466066 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 04:37:47.468214 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.452621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.19.92.60:41653)
0809 04:37:47.453073 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.453094 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.453137 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:37:47.454014 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.454039 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.454478 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.454875 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.455330 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.456830 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.458213 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.458919 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.459564 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.461335 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.461341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.461359 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.461391 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.464011 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.464578 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.464584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.464590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.464731 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.465045 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.465049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.465051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.465590 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.466584 (+ 994us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.466767 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.467120 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260809 04:37:47.468583 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.457520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56138)
0809 04:37:47.457775 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.457789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:47.457866 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:47.458546 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:47.458549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:47.458742 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:47.458853 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.458861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.459949 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.459953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.460783 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.460796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.461547 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.461550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.461617 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.462229 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:47.462243 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.463713 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.466987 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.466991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.466993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.467246 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.467545 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.467548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.467550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.467636 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.468097 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.468100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.468203 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.468279 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.468420 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260809 04:37:47.469249 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.458001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.19.92.60:41653)
0809 04:37:47.458263 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.458272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.458285 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:37:47.458318 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.458321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.458359 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.458990 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.459074 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.459830 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.460958 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.461402 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.461814 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.463865 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.463868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.463870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.463898 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.466866 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.467357 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.467360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.467362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.467437 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.467745 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.467748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.467750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.467981 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.468856 (+ 875us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.468929 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.469042 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
I20260809 04:37:47.469473 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.456601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.19.92.60:41653)
0809 04:37:47.457009 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.457033 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.457339 (+ 306us) server_negotiation.cc:407] Connection header received
0809 04:37:47.457517 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.457520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.457564 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.457618 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.458670 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.459663 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.460961 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.461394 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.462339 (+ 945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.465553 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.465557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.465560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.465579 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.467276 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.467764 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.467766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.467768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.467850 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.468161 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.468164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.468166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.468367 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.469083 (+ 716us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.469210 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.469321 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":337,"thread_start_us":146,"threads_started":1}
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:47.472997 7696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4: Bootstrap starting.
I20260809 04:37:47.473035 7572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb: Bootstrap starting.
I20260809 04:37:47.475589 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.462026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.19.92.62:45579)
0809 04:37:47.462150 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.462154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.462736 (+ 582us) server_negotiation.cc:407] Connection header received
0809 04:37:47.462782 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.462785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.462831 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.462911 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.465103 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.465835 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.467323 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.467638 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.467705 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.471926 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.471931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.471934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.471962 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.473564 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.474138 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.474142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.474144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.474253 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.474552 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.474555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.474557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.474814 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.474914 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.475291 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.475420 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:37:47.475629 7572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:47.475632 7696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:47.475608 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.461780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:45579 (local address 127.0.0.1:60958)
0809 04:37:47.462283 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.462337 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:47.462453 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:47.462958 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:47.462964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:47.464841 (+ 1877us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:47.464991 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.464998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.465967 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.465971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.467190 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.467201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.467838 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.467841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.467982 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.470144 (+ 2162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:47.470158 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.471767 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.473731 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.473741 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.473747 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.474030 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.474351 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.474354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.474356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.474449 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.474927 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.474934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.475066 (+ 132us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.475219 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.475422 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260809 04:37:47.476449 7572 log.cc:824] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:47.476451 7696 log.cc:824] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:47.477303 7825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:37:47.479158 7572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb: No bootstrap required, opened a new log
I20260809 04:37:47.479342 7696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4: No bootstrap required, opened a new log
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
I20260809 04:37:47.483265 7696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.483294 7572 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.483485 7696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:47.483501 7572 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:47.483539 7696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10fe7a9e7079469e8c545522df7e49c4, State: Initialized, Role: FOLLOWER
I20260809 04:37:47.483546 7572 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcea1ef6c46e4d32942eedeaff6ca9fb, State: Initialized, Role: FOLLOWER
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:47.484061 7572 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.484061 7696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.485445 7837 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.485585 7696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:47.486464 7837 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:37:47.485589 7572 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:47.488026 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.477793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:39459 (local address 127.0.0.1:51944)
0809 04:37:47.477953 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.477967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:47.478033 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:47.478463 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:47.478466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:47.478663 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:47.478782 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.478791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.479804 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.479807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.480632 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.480643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.481368 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.481372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.481461 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.482133 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:47.482147 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.484205 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.486334 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.486336 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.486337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.486577 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.486888 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.486890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.486892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.486994 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.487502 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.487505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.487617 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.487716 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.487859 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 04:37:47.488564 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.478033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.19.92.61:39459)
0809 04:37:47.478160 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.478165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.478178 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:37:47.478223 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.478226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.478269 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.478347 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.478942 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.479682 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.480764 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.481106 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.481243 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.484365 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.484369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.484371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.484399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.486221 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.486693 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.486697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.486699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.486788 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.487095 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.487099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.487100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.487361 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.487455 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.488299 (+ 844us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.488420 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:37:47.489635 7584 master_runner.cc:186] Error getting master registration for 127.19.92.62:45579: OK, instance_id { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" instance_seqno: 1786250266977087 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260809 04:37:47.491843 7860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:37:47.491915 7860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:37:47.491978 7825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:37:47.492219 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.492332 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:37:47.489173 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.477460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.19.92.62:45579)
0809 04:37:47.477650 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.477654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.477665 (+ 11us) server_negotiation.cc:407] Connection header received
0809 04:37:47.477697 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.477700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.477740 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.477804 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.478929 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.479644 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.480790 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.481112 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.481225 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.484574 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.484578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.484581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.484606 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.486322 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.486793 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.486797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.486798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.486880 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.487199 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.487203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.487204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.487401 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.487564 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.488894 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.489006 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260809 04:37:47.491916 7584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 04:37:47.495498 7868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:37:47.495609 7868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
I20260809 04:37:47.500490 7459 master_runner.cc:186] Error getting master registration for 127.19.92.60:41653: OK, instance_id { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" instance_seqno: 1786250267447409 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:37:47.503849 7459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:37:47.504840 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.494545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.19.92.61:39459)
0809 04:37:47.494643 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.494647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.494660 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:37:47.494692 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.494694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.494731 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.494807 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.495645 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.496377 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.497543 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.497874 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.497969 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.500961 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.500965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.500969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.501007 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.502621 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.503122 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.503126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.503128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.503213 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.503509 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.503513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.503515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.503730 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.503910 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.504566 (+ 656us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.504682 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:37:47.508491 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 09 04:37:47 dist-test-slave-rmhg krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
I20260809 04:37:47.512475 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.492443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:41653 (local address 127.0.0.1:56144)
0809 04:37:47.492767 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:47.492780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:47.492849 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:47.503989 (+ 11140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:47.503994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:47.504204 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:47.504322 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.504330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.505631 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.505635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.506475 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:47.506485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.507146 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.507149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:47.507225 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.507766 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:47.507781 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:47.509295 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:47.511109 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.511112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.511114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.511329 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.511556 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:47.511558 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:47.511560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:47.511624 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:47.512012 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:47.512015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:47.512091 (+ 76us) client_negotiation.cc:769] Sending connection context
0809 04:37:47.512180 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 04:37:47.512309 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
I20260809 04:37:47.512904 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.492673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.19.92.60:41653)
0809 04:37:47.493071 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.493075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.493088 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:37:47.493119 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.493122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.493203 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.493272 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.504618 (+ 11346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.505524 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.506596 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.506935 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.507013 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.509420 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.509423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.509426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.509446 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.511002 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.511414 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.511417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.511418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.511478 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.511696 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.511698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.511700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.511886 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.511962 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.512671 (+ 709us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.512779 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260809 04:37:47.516346 7825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f: Bootstrap starting.
I20260809 04:37:47.519480 7825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:47.520444 7825 log.cc:824] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:47.524675 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.510951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.19.92.61:39459)
0809 04:37:47.511078 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.511082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.512893 (+ 1811us) server_negotiation.cc:407] Connection header received
0809 04:37:47.513319 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.513322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.513372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.513418 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.514530 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.515332 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.518086 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.518463 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.518545 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.521122 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.521126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.521129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.521192 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.522782 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.523276 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.523280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.523283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.523366 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.523650 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.523654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.523656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.523866 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.523958 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.524366 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.524508 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:37:47.528452 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.511678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.19.92.62:45579)
0809 04:37:47.511791 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.511795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.516768 (+ 4973us) server_negotiation.cc:407] Connection header received
0809 04:37:47.516945 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.516948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.517001 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.517073 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.519152 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.519936 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.522676 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.523070 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.523152 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.524757 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.524761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.524771 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.524801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.526637 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.527115 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.527119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.527121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.527209 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.527484 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.527487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.527488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.527701 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.527818 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.528165 (+ 347us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.528289 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:37:47.535480 7825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f: No bootstrap required, opened a new log
I20260809 04:37:47.540299 7825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.540594 7825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:47.540650 7825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e92d70271c48769a4d05aff4e8585f, State: Initialized, Role: FOLLOWER
I20260809 04:37:47.540522 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:47.512290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.19.92.60:41653)
0809 04:37:47.512393 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:47.512397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:47.522192 (+ 9795us) server_negotiation.cc:407] Connection header received
0809 04:37:47.522238 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:47.522241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:47.522287 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:47.522356 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:47.525396 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.526172 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.534189 (+ 8017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:47.534550 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:47.534638 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:47.536563 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:47.536567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:47.536571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:47.536597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:47.538208 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.538722 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.538725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.538727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.538811 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:47.539306 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:47.539309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:47.539311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:47.539549 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:47.539685 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:47.540217 (+ 532us) server_negotiation.cc:299] Negotiation successful
0809 04:37:47.540355 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 04:37:47.541289 7825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.542822 7825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:47.543080 7882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.543313 7882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [sys.catalog]: This master's current role is: FOLLOWER
W20260809 04:37:47.550604 7901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:37:47.550716 7901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:37:47.551611 7709 master_runner.cc:263] No leader master found from master 44e92d70271c48769a4d05aff4e8585f
I20260809 04:37:47.551693 7709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:37:47.625408 7837 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:37:47.625624 7837 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.626866 7837 leader_election.cc:288] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10fe7a9e7079469e8c545522df7e49c4 (127.19.92.61:39459), 44e92d70271c48769a4d05aff4e8585f (127.19.92.60:41653)
I20260809 04:37:47.627640 7785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e92d70271c48769a4d05aff4e8585f" is_pre_election: true
I20260809 04:37:47.628068 7785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcea1ef6c46e4d32942eedeaff6ca9fb in term 0.
I20260809 04:37:47.628893 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10fe7a9e7079469e8c545522df7e49c4" is_pre_election: true
I20260809 04:37:47.629281 7652 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcea1ef6c46e4d32942eedeaff6ca9fb in term 0.
I20260809 04:37:47.629807 7481 leader_election.cc:302] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44e92d70271c48769a4d05aff4e8585f, bcea1ef6c46e4d32942eedeaff6ca9fb; no voters:
I20260809 04:37:47.630554 7837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:37:47.630658 7837 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:37:47.630716 7837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:47.632187 7837 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.633073 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10fe7a9e7079469e8c545522df7e49c4"
I20260809 04:37:47.633258 7652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:47.634724 7652 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcea1ef6c46e4d32942eedeaff6ca9fb in term 1.
I20260809 04:37:47.635128 7483 leader_election.cc:302] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10fe7a9e7079469e8c545522df7e49c4, bcea1ef6c46e4d32942eedeaff6ca9fb; no voters:
I20260809 04:37:47.636013 7785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e92d70271c48769a4d05aff4e8585f"
I20260809 04:37:47.636189 7785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:47.637776 7785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcea1ef6c46e4d32942eedeaff6ca9fb in term 1.
I20260809 04:37:47.638228 7837 leader_election.cc:288] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [CANDIDATE]: Term 1 election: Requested vote from peers 10fe7a9e7079469e8c545522df7e49c4 (127.19.92.61:39459), 44e92d70271c48769a4d05aff4e8585f (127.19.92.60:41653)
I20260809 04:37:47.638427 7837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:37:47.638532 7837 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [term 1 LEADER]: Becoming Leader. State: Replica: bcea1ef6c46e4d32942eedeaff6ca9fb, State: Running, Role: LEADER
I20260809 04:37:47.638954 7837 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } }
I20260809 04:37:47.646106 7837 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcea1ef6c46e4d32942eedeaff6ca9fb. Latest consensus state: current_term: 1 leader_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.646265 7837 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:47.646804 7907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:37:47.647924 7907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:37:47.655117 7785 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [term 1 FOLLOWER]: Refusing update from remote peer bcea1ef6c46e4d32942eedeaff6ca9fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:37:47.655817 7903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:37:47.662791 7882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcea1ef6c46e4d32942eedeaff6ca9fb. Latest consensus state: current_term: 1 leader_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.662920 7882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:37:47.664724 7652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [term 1 FOLLOWER]: Refusing update from remote peer bcea1ef6c46e4d32942eedeaff6ca9fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:37:47.665436 7903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:37:47.669713 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcea1ef6c46e4d32942eedeaff6ca9fb. Latest consensus state: current_term: 1 leader_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.669842 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:37:47.676332 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.676489 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:47.681001 7882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.681171 7882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:37:47.681624 7908 mvcc.cc:205] Tried to move back new op lower bound from 7316481096307388416 to 7316481096267034624. Current Snapshot: MvccSnapshot[applied={T|T < 7316481096307388416}]
I20260809 04:37:47.682981 7907 catalog_manager.cc:1347] Generated new cluster ID: fbd5612a28d94782a10d477753b97674
I20260809 04:37:47.683050 7907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:37:47.683312 7909 mvcc.cc:205] Tried to move back new op lower bound from 7316481096307388416 to 7316481096267034624. Current Snapshot: MvccSnapshot[applied={T|T < 7316481096307388416}]
I20260809 04:37:47.685076 7837 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.685230 7837 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:47.686131 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcea1ef6c46e4d32942eedeaff6ca9fb" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45579 } } peers { permanent_uuid: "10fe7a9e7079469e8c545522df7e49c4" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 39459 } } peers { permanent_uuid: "44e92d70271c48769a4d05aff4e8585f" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 41653 } } }
I20260809 04:37:47.686246 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:37:47.717463 7907 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:37:47.718819 7907 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:37:47.725637 7907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcea1ef6c46e4d32942eedeaff6ca9fb: Generated new TSK 0
I20260809 04:37:47.726517 7907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:37:48.493273 7860 catalog_manager.cc:1259] Loaded cluster ID: fbd5612a28d94782a10d477753b97674
I20260809 04:37:48.493358 7860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4: loading cluster ID for follower catalog manager: success
I20260809 04:37:48.495034 7860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4: acquiring CA information for follower catalog manager: success
I20260809 04:37:48.496287 7860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10fe7a9e7079469e8c545522df7e49c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:37:48.552074 7901 catalog_manager.cc:1259] Loaded cluster ID: fbd5612a28d94782a10d477753b97674
I20260809 04:37:48.552160 7901 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f: loading cluster ID for follower catalog manager: success
I20260809 04:37:48.553933 7901 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f: acquiring CA information for follower catalog manager: success
I20260809 04:37:48.555202 7901 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44e92d70271c48769a4d05aff4e8585f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:37:48.780874 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 7459
I20260809 04:37:48.790836 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 7584
I20260809 04:37:48.807214 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 7709
2026-08-09T04:37:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3281 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:37:48 dist-test-slave-rmhg krb5kdc[7928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:37:48 dist-test-slave-rmhg krb5kdc[7928](info): set up 2 sockets
Aug 09 04:37:48 dist-test-slave-rmhg krb5kdc[7928](info): commencing operation
krb5kdc: starting...
W20260809 04:37:49.941596 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:37:50 dist-test-slave-rmhg krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:37:50.067763 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:45525
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:50.181211 7948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:50.181486 7948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:50.181541 7948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:50.185494 7948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:50.185549 7948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:50.185575 7948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:50.185596 7948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:50.185622 7948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:50.190871 7948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:45525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:50.192008 7948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:50.193115 7948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:50.202046 7957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:50.202051 7956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:50.202051 7960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:50.202360 7948 server_base.cc:1034] running on GCE node
I20260809 04:37:50.202961 7948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:50.204154 7948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:50.205329 7948 hybrid_clock.cc:648] HybridClock initialized: now 1786250270205310 us; error 30 us; skew 500 ppm
Aug 09 04:37:50 dist-test-slave-rmhg krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250270, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:50.208072 7948 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:37:50.209041 7948 webserver.cc:460] Webserver started at http://127.19.92.62:43769/ using document root <none> and password file <none>
I20260809 04:37:50.209625 7948 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:50.209687 7948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:50.209894 7948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:37:50.211984 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "564af804614b4de6ab8e011a3317f90c"
format_stamp: "Formatted at 2026-08-09 04:37:50 on dist-test-slave-rmhg"
server_key: "f9e8268d528c8e2d586c799cb132a033"
server_key_iv: "6159c1340b1ce75986298604f0684229"
server_key_version: "encryptionkey@0"
I20260809 04:37:50.212546 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "564af804614b4de6ab8e011a3317f90c"
format_stamp: "Formatted at 2026-08-09 04:37:50 on dist-test-slave-rmhg"
server_key: "f9e8268d528c8e2d586c799cb132a033"
server_key_iv: "6159c1340b1ce75986298604f0684229"
server_key_version: "encryptionkey@0"
I20260809 04:37:50.216325 7948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:37:50.218947 7977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:50.220276 7948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:37:50.220412 7948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "564af804614b4de6ab8e011a3317f90c"
format_stamp: "Formatted at 2026-08-09 04:37:50 on dist-test-slave-rmhg"
server_key: "f9e8268d528c8e2d586c799cb132a033"
server_key_iv: "6159c1340b1ce75986298604f0684229"
server_key_version: "encryptionkey@0"
I20260809 04:37:50.220554 7948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:50.242390 7948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:50.245788 7948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:50.245959 7948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:50.252943 7948 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:45525
I20260809 04:37:50.252964 8060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:45525 every 8 connection(s)
I20260809 04:37:50.253938 7948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:37:50.255183 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 7948
I20260809 04:37:50.255283 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:37:50.255563 19824 external_mini_cluster.cc:1428] Setting key d3c20ca778a6a407724653b69b188a19
I20260809 04:37:50.257195 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:50.263927 8061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c: Bootstrap starting.
Aug 09 04:37:50 dist-test-slave-rmhg krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:50.266343 8061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:50.267068 8061 log.cc:824] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:50.268934 8061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c: No bootstrap required, opened a new log
I20260809 04:37:50.270040 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:50.256954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.19.92.62:45525)
0809 04:37:50.257651 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:50.257673 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:50.257717 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:37:50.258498 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:50.258524 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:50.258989 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:50.259316 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:50.260209 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:50.261232 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:50.262346 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:50.262812 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:50.262923 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:50.265634 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:50.265639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:50.265652 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:50.265678 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:50.267300 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:50.267719 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:50.267724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:50.267728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:50.267840 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:50.268107 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:50.268110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:50.268112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:50.268520 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:50.268695 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:50.268983 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 04:37:50.269321 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":183,"threads_started":1}
I20260809 04:37:50.272497 8061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564af804614b4de6ab8e011a3317f90c" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45525 } }
I20260809 04:37:50.272742 8061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:50.272805 8061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564af804614b4de6ab8e011a3317f90c, State: Initialized, Role: FOLLOWER
I20260809 04:37:50.272881 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:37:50.273296 8061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564af804614b4de6ab8e011a3317f90c" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45525 } }
I20260809 04:37:50.273470 8061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:37:50.273535 8061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:37:50.273633 8061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:50.274523 8061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564af804614b4de6ab8e011a3317f90c" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45525 } }
I20260809 04:37:50.274848 8061 leader_election.cc:302] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 564af804614b4de6ab8e011a3317f90c; no voters:
I20260809 04:37:50.275125 8061 leader_election.cc:288] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:37:50.275249 8066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:37:50.275447 8066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [term 1 LEADER]: Becoming Leader. State: Replica: 564af804614b4de6ab8e011a3317f90c, State: Running, Role: LEADER
I20260809 04:37:50.275799 8066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564af804614b4de6ab8e011a3317f90c" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45525 } }
I20260809 04:37:50.276338 8061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:50.277441 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 564af804614b4de6ab8e011a3317f90c. Latest consensus state: current_term: 1 leader_uuid: "564af804614b4de6ab8e011a3317f90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564af804614b4de6ab8e011a3317f90c" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45525 } } }
I20260809 04:37:50.277578 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:50.277923 8067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "564af804614b4de6ab8e011a3317f90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564af804614b4de6ab8e011a3317f90c" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45525 } } }
I20260809 04:37:50.278065 8067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:50.277983 8079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:37:50.280952 8079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:37:50.286633 8079 catalog_manager.cc:1347] Generated new cluster ID: d3e9c0fb83ff4b1d82489f81c438ccd1
I20260809 04:37:50.286702 8079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:37:50.311195 8079 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:37:50.312565 8079 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:37:50.323588 8079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 564af804614b4de6ab8e011a3317f90c: Generated new TSK 0
I20260809 04:37:50.324561 8079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:37:50.328532 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 7948
2026-08-09T04:37:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1525 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:37:50 dist-test-slave-rmhg krb5kdc[8097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:37:50 dist-test-slave-rmhg krb5kdc[8097](info): set up 2 sockets
Aug 09 04:37:50 dist-test-slave-rmhg krb5kdc[8097](info): commencing operation
krb5kdc: starting...
W20260809 04:37:51.468016 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:37:51 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:37:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:37:51.600101 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41143
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:51.717127 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:51.717506 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:51.717569 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:51.721799 8118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:51.721887 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:51.721916 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:51.721938 8118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:51.721966 8118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:51.727610 8118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:51.728812 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:51.729960 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:51.738471 8127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:51.738514 8133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:51.738638 8118 server_base.cc:1034] running on GCE node
W20260809 04:37:51.738474 8126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:51.739358 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:51.740458 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:51.741611 8118 hybrid_clock.cc:648] HybridClock initialized: now 1786250271741590 us; error 33 us; skew 500 ppm
Aug 09 04:37:51 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:51.744267 8118 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:37:51.745253 8118 webserver.cc:460] Webserver started at http://127.19.92.62:40895/ using document root <none> and password file <none>
I20260809 04:37:51.745810 8118 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:51.745880 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:51.746093 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:37:51.748240 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "cf7cca8eed144dd2870a19f74e6c7797"
format_stamp: "Formatted at 2026-08-09 04:37:51 on dist-test-slave-rmhg"
server_key: "9f8975f5d9a6aae617a850e79dcd3525"
server_key_iv: "f5267fb9e33380acd3a7041b1f6b053f"
server_key_version: "encryptionkey@0"
I20260809 04:37:51.748793 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "cf7cca8eed144dd2870a19f74e6c7797"
format_stamp: "Formatted at 2026-08-09 04:37:51 on dist-test-slave-rmhg"
server_key: "9f8975f5d9a6aae617a850e79dcd3525"
server_key_iv: "f5267fb9e33380acd3a7041b1f6b053f"
server_key_version: "encryptionkey@0"
I20260809 04:37:51.752596 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:37:51.755383 8143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:51.756798 8118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:37:51.756933 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "cf7cca8eed144dd2870a19f74e6c7797"
format_stamp: "Formatted at 2026-08-09 04:37:51 on dist-test-slave-rmhg"
server_key: "9f8975f5d9a6aae617a850e79dcd3525"
server_key_iv: "f5267fb9e33380acd3a7041b1f6b053f"
server_key_version: "encryptionkey@0"
I20260809 04:37:51.757053 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:51.780750 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:51.784058 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:51.784258 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:51.791541 8118 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41143
I20260809 04:37:51.791797 8233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41143 every 8 connection(s)
I20260809 04:37:51.792630 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:37:51.795740 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:51.798282 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 8118
I20260809 04:37:51.798403 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:37:51.798653 19824 external_mini_cluster.cc:1428] Setting key b5a35fdff38c80cc3d827acdb7e71f0f
I20260809 04:37:51.802739 8235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Bootstrap starting.
I20260809 04:37:51.805593 8235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:51.806545 8235 log.cc:824] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:37:51 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:51.809020 8235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: No bootstrap required, opened a new log
I20260809 04:37:51.813115 8235 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:51.813455 8235 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:51.813524 8235 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7cca8eed144dd2870a19f74e6c7797, State: Initialized, Role: FOLLOWER
I20260809 04:37:51.813971 8235 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:51.814107 8235 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:37:51.814173 8235 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:37:51.814268 8235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:51.814385 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:51.800116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.19.92.62:41143)
0809 04:37:51.800599 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:51.800624 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:51.800670 (+ 46us) server_negotiation.cc:407] Connection header received
0809 04:37:51.801520 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:51.801546 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:51.802018 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:51.802408 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:51.803162 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:51.804693 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:51.805731 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:51.806441 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:51.806607 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:51.809576 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:51.809582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:51.809597 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:51.809627 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:51.811285 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:51.811816 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:51.811820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:51.811825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:51.811941 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:51.812214 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:51.812216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:51.812217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:51.812638 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:51.812796 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:51.813079 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 04:37:51.813401 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260809 04:37:51.815233 8235 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:51.815583 8235 leader_election.cc:302] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf7cca8eed144dd2870a19f74e6c7797; no voters:
I20260809 04:37:51.815927 8235 leader_election.cc:288] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:37:51.816021 8243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:37:51.816197 8243 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 1 LEADER]: Becoming Leader. State: Replica: cf7cca8eed144dd2870a19f74e6c7797, State: Running, Role: LEADER
I20260809 04:37:51.816589 8243 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:51.817345 8235 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:51.818601 8244 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf7cca8eed144dd2870a19f74e6c7797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } } }
I20260809 04:37:51.818735 8244 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:51.819001 8245 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf7cca8eed144dd2870a19f74e6c7797. Latest consensus state: current_term: 1 leader_uuid: "cf7cca8eed144dd2870a19f74e6c7797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } } }
I20260809 04:37:51.819097 8245 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:51.819134 8253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:37:51.822320 8253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:37:51.828225 8253 catalog_manager.cc:1347] Generated new cluster ID: dc3cede3202c4593af7b49b1743dee88
I20260809 04:37:51.828305 8253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:37:51.864872 8253 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:37:51.865907 8253 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:37:51.871788 8253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Generated new TSK 0
I20260809 04:37:51.872471 8253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:37:51.909741 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41143
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:52.044924 8269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:52.045223 8269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:52.045267 8269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:52.049245 8269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:52.049307 8269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:52.049366 8269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:37:52.054546 8269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:52.055554 8269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:52.056514 8269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:37:52.065985 8269 server_base.cc:1034] running on GCE node
W20260809 04:37:52.065936 8276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:52.065956 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:52.066134 8275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:52.066560 8269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:52.067176 8269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:52.068342 8269 hybrid_clock.cc:648] HybridClock initialized: now 1786250272068325 us; error 26 us; skew 500 ppm
Aug 09 04:37:52 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250272, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:52.071048 8269 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:37:52.071921 8269 webserver.cc:460] Webserver started at http://127.19.92.1:38569/ using document root <none> and password file <none>
I20260809 04:37:52.072481 8269 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:52.072548 8269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:52.072767 8269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:37:52.074862 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "17cd6219ec254798858cbc9739ae5519"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ad924b58b1ed082076ea91a93404b300"
server_key_iv: "491afa55c472e82878303972115f2dec"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.075390 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "17cd6219ec254798858cbc9739ae5519"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ad924b58b1ed082076ea91a93404b300"
server_key_iv: "491afa55c472e82878303972115f2dec"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.079324 8269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 04:37:52.081872 8292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.083130 8269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:37:52.083250 8269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "17cd6219ec254798858cbc9739ae5519"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ad924b58b1ed082076ea91a93404b300"
server_key_iv: "491afa55c472e82878303972115f2dec"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.083349 8269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:52.100806 8269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:52.103853 8269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:52.104027 8269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:52.104581 8269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:52.105449 8269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:52.105499 8269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.105537 8269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:52.105557 8269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.113867 8269 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:44411
I20260809 04:37:52.113905 8459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:44411 every 8 connection(s)
I20260809 04:37:52.114773 8269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:37:52.118355 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 8269
I20260809 04:37:52.118496 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:37:52.118769 19824 external_mini_cluster.cc:1428] Setting key 87b861729bc7220a5cc0bb831e2e992a
Aug 09 04:37:52 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250272, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:37:52.129845 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.116631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:35723 (local address 127.19.92.62:41143)
0809 04:37:52.116812 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.116816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.117769 (+ 953us) server_negotiation.cc:407] Connection header received
0809 04:37:52.119401 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.119404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.119489 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.119580 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:52.121093 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.121683 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.122872 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.123106 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.123197 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.126048 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:52.126051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:52.126053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:52.126079 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:52.127492 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:52.128113 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:52.128116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:52.128117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:52.128202 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:52.128604 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:52.128606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:52.128607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:52.128802 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:52.128879 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.129586 (+ 707us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.129715 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 04:37:52.130780 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.117116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:35723)
0809 04:37:52.117614 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.117660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.119171 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.119795 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.119806 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.120364 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:52.120930 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.120946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.121811 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.121815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.122746 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.122756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.123252 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.123256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.123438 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.124011 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:52.124028 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:52.125855 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:52.127609 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:52.127616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:52.127634 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:52.127987 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:52.128298 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:52.128301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:52.128303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:52.128507 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:52.128910 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:52.128919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:52.129285 (+ 366us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.129527 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.129861 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260809 04:37:52.132128 8461 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:37:52.132401 8461 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:52.132946 8461 heartbeater.cc:511] Master 127.19.92.62:41143 requested a full tablet report, sending...
I20260809 04:37:52.134527 8165 ts_manager.cc:194] Registered new tserver with Master: 17cd6219ec254798858cbc9739ae5519 (127.19.92.1:44411)
I20260809 04:37:52.135907 8165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:35723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:37:52.148447 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41143
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:52.261968 8469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:52.262228 8469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:52.262272 8469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:52.266258 8469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:52.266319 8469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:52.266376 8469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:37:52.271677 8469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:52.272775 8469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:52.273802 8469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:52.283812 8480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:52.283813 8482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:52.283813 8485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:52.283885 8469 server_base.cc:1034] running on GCE node
I20260809 04:37:52.284493 8469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:52.285172 8469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:52.286358 8469 hybrid_clock.cc:648] HybridClock initialized: now 1786250272286332 us; error 39 us; skew 500 ppm
Aug 09 04:37:52 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250272, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:52.289289 8469 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:37:52.290213 8469 webserver.cc:460] Webserver started at http://127.19.92.2:39655/ using document root <none> and password file <none>
I20260809 04:37:52.290750 8469 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:52.290808 8469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:52.290983 8469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:37:52.292961 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "e7613783bd524ea1851327bc707711e8"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ca44a4651a35dfd56eb557e1ed95da89"
server_key_iv: "fabe24e38cc7a23520b4378c2e4425d0"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.293526 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "e7613783bd524ea1851327bc707711e8"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ca44a4651a35dfd56eb557e1ed95da89"
server_key_iv: "fabe24e38cc7a23520b4378c2e4425d0"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.297127 8469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:37:52.299516 8497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.300809 8469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:37:52.300936 8469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "e7613783bd524ea1851327bc707711e8"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ca44a4651a35dfd56eb557e1ed95da89"
server_key_iv: "fabe24e38cc7a23520b4378c2e4425d0"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.301050 8469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:52.314178 8469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:52.317663 8469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:52.317862 8469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:52.318593 8469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:52.319536 8469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:52.319587 8469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.319630 8469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:52.319659 8469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.328790 8469 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:33223
I20260809 04:37:52.328820 8685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:33223 every 8 connection(s)
I20260809 04:37:52.329829 8469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:37:52.336225 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 8469
I20260809 04:37:52.336323 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:37:52.336573 19824 external_mini_cluster.cc:1428] Setting key e06e8e4f301ff5ff449f7dcbc7bff0a3
Aug 09 04:37:52 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250272, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:52.344841 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.331769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:34927 (local address 127.19.92.62:41143)
0809 04:37:52.331935 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.331939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.332793 (+ 854us) server_negotiation.cc:407] Connection header received
0809 04:37:52.334265 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.334269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.334331 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.334404 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:52.335693 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.336235 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.337611 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.337901 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.338032 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.341060 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:52.341064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:52.341066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:52.341092 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:52.342606 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:52.343186 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:52.343190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:52.343192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:52.343286 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:52.343641 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:52.343644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:52.343646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:52.343902 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:52.344006 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.344530 (+ 524us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.344667 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:37:52.345695 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.332160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:34927)
0809 04:37:52.332628 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.332674 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.334027 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.334603 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.334610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.335010 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:52.335522 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.335536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.336399 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.336402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.337492 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.337502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.338069 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.338073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.338281 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.338921 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:52.338943 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:52.340837 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:52.342738 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:52.342745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:52.342760 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:52.343082 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:52.343373 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:52.343376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:52.343377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:52.343546 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:52.344001 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:52.344007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:52.344273 (+ 266us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.344454 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.344760 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260809 04:37:52.347026 8687 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41143
I20260809 04:37:52.347277 8687 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:37:52.347857 8687 heartbeater.cc:511] Master 127.19.92.62:41143 requested a full tablet report, sending...
I20260809 04:37:52.349007 8165 ts_manager.cc:194] Registered new tserver with Master: e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
I20260809 04:37:52.349669 8165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:34927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:37:52.368526 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41143
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:52.487716 8695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:52.488071 8695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:52.488157 8695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:52.492271 8695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:52.492336 8695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:52.492420 8695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:37:52.498051 8695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:52.499188 8695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:52.500293 8695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:52.510157 8707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:52.510123 8706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:52.510286 8695 server_base.cc:1034] running on GCE node
W20260809 04:37:52.510313 8709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:52.510862 8695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:52.511507 8695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:52.512676 8695 hybrid_clock.cc:648] HybridClock initialized: now 1786250272512656 us; error 33 us; skew 500 ppm
Aug 09 04:37:52 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250272, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:52.518644 8695 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:37:52.519708 8695 webserver.cc:460] Webserver started at http://127.19.92.3:35143/ using document root <none> and password file <none>
I20260809 04:37:52.520370 8695 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:52.520448 8695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:52.520673 8695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:37:52.522904 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "ce2735617de1401987f7556909143af3"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "bd6cee0ce9628662802031e4499c5def"
server_key_iv: "d10153caca78480f010ac539c3412a59"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.523505 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "ce2735617de1401987f7556909143af3"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "bd6cee0ce9628662802031e4499c5def"
server_key_iv: "d10153caca78480f010ac539c3412a59"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.527535 8695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:37:52.530153 8721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.531473 8695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:37:52.531603 8695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "ce2735617de1401987f7556909143af3"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "bd6cee0ce9628662802031e4499c5def"
server_key_iv: "d10153caca78480f010ac539c3412a59"
server_key_version: "encryptionkey@0"
I20260809 04:37:52.531714 8695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:52.551342 8695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:52.554641 8695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:52.554811 8695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:52.555408 8695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:52.556262 8695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:52.556309 8695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.556349 8695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:52.556370 8695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:52.565598 8695 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:42307
I20260809 04:37:52.565663 8907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:42307 every 8 connection(s)
I20260809 04:37:52.566578 8695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
Aug 09 04:37:52 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250272, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:52.576643 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 8695
I20260809 04:37:52.576768 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:37:52.577031 19824 external_mini_cluster.cc:1428] Setting key 9746c426c348ac48aa0a1bce63b677c5
I20260809 04:37:52.580910 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.568610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:59407 (local address 127.19.92.62:41143)
0809 04:37:52.568779 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.568782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.569765 (+ 983us) server_negotiation.cc:407] Connection header received
0809 04:37:52.571180 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.571183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.571234 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.571313 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:52.572608 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.573182 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.574436 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.574674 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.574755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.577351 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:52.577354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:52.577356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:52.577381 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:52.578810 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:52.579380 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:52.579384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:52.579386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:52.579474 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:52.579815 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:52.579818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:52.579820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:52.580015 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:52.580100 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.580632 (+ 532us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.580754 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 04:37:52.581858 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.569015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:59407)
0809 04:37:52.569600 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.569648 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.570942 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.571513 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.571520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.571907 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:52.572447 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.572464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.573334 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.573337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.574313 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.574325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.574802 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.574805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.574958 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.575561 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:52.575579 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:52.577140 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:52.578922 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:52.578929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:52.578947 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:52.579290 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:52.579558 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:52.579562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:52.579564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:52.579726 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:52.580119 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:52.580127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:52.580398 (+ 271us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.580575 (+ 177us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.580931 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":160,"threads_started":1}
I20260809 04:37:52.583040 8908 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41143
I20260809 04:37:52.583302 8908 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:52.583818 8908 heartbeater.cc:511] Master 127.19.92.62:41143 requested a full tablet report, sending...
I20260809 04:37:52.584899 8165 ts_manager.cc:194] Registered new tserver with Master: ce2735617de1401987f7556909143af3 (127.19.92.3:42307)
I20260809 04:37:52.585570 8165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:59407
I20260809 04:37:52.591024 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:37:52.600630 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.593094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.19.92.62:41143)
0809 04:37:52.593268 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.593272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.593402 (+ 130us) server_negotiation.cc:407] Connection header received
0809 04:37:52.593538 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.593541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.593587 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.593652 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:52.594394 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.594940 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.595766 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.595987 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.596081 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.597567 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:52.597571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:52.597573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:52.597600 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:52.598894 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:52.599323 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:52.599335 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:52.599337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:52.599421 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:52.599688 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:52.599692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:52.599694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:52.599945 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:52.600072 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.600332 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.600458 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 04:37:52.603675 8165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:37:52.606842 8165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:37:52.621445 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.615018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:44411 (local address 127.0.0.1:57170)
0809 04:37:52.615653 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.615664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.615732 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.616640 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.616646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.616671 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:52.616819 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.616826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.618415 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.618418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.619682 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.619693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.620770 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.620774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.620946 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.621007 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.621119 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.621288 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":77,"thread_start_us":105,"threads_started":1}
I20260809 04:37:52.621495 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.615594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.19.92.1:44411)
0809 04:37:52.616109 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.616126 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.616152 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:37:52.616231 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.616236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.616453 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.616632 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:52.616960 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.618294 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.619810 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.620648 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.620752 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.620901 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.621182 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.621288 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":100,"threads_started":1}
I20260809 04:37:52.621757 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.616206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.19.92.2:33223)
0809 04:37:52.616696 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.616713 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.616734 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:37:52.616804 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.616809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.617007 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.617145 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:52.617415 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.618476 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.619823 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.620648 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.620752 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.620901 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.621519 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.621615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":116,"threads_started":1}
I20260809 04:37:52.622012 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.616040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33223 (local address 127.0.0.1:40662)
0809 04:37:52.616337 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.616351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.616459 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.617207 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.617210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.617230 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:52.617309 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.617315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.618752 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.618756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.619697 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.619707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.620770 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.620774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.621267 (+ 493us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.621290 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.621362 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.621428 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":332,"thread_start_us":90,"threads_started":1}
I20260809 04:37:52.622237 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.614969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:42307 (local address 127.0.0.1:53334)
0809 04:37:52.615551 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.615604 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.615733 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.616903 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.616906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.616924 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:52.617027 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.617034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.618124 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.618127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.619682 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.619693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.621896 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.621900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.621986 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.622008 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.622055 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.622121 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":27,"thread_start_us":81,"threads_started":1}
I20260809 04:37:52.622951 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.615089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.19.92.3:42307)
0809 04:37:52.616466 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.616488 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.616511 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:37:52.616586 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.616591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.616769 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.616908 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:52.617133 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.618003 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.620930 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.621763 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.622527 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.622673 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.622721 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.622807 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1274,"thread_start_us":180,"threads_started":1}
I20260809 04:37:52.624743 8365 tablet_service.cc:1467] Processing CreateTablet for tablet daf267bcbefb43269ef6aee8aca49122 (DEFAULT_TABLE table=test-table [id=18031a2fdecc4278a6aa9e09703d0bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:52.624843 8579 tablet_service.cc:1467] Processing CreateTablet for tablet daf267bcbefb43269ef6aee8aca49122 (DEFAULT_TABLE table=test-table [id=18031a2fdecc4278a6aa9e09703d0bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:52.625777 8365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf267bcbefb43269ef6aee8aca49122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:52.625435 8798 tablet_service.cc:1467] Processing CreateTablet for tablet daf267bcbefb43269ef6aee8aca49122 (DEFAULT_TABLE table=test-table [id=18031a2fdecc4278a6aa9e09703d0bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:52.625924 8579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf267bcbefb43269ef6aee8aca49122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:52.626385 8798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf267bcbefb43269ef6aee8aca49122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:52.631819 8935 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Bootstrap starting.
I20260809 04:37:52.633458 8939 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Bootstrap starting.
I20260809 04:37:52.634043 8935 tablet_bootstrap.cc:654] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:52.634742 8935 log.cc:824] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:52.635229 8939 tablet_bootstrap.cc:654] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:52.635406 8936 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Bootstrap starting.
I20260809 04:37:52.636445 8939 log.cc:824] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:52.637226 8935 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: No bootstrap required, opened a new log
I20260809 04:37:52.637424 8935 ts_tablet_manager.cc:1397] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:37:52.637521 8936 tablet_bootstrap.cc:654] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:52.638214 8939 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: No bootstrap required, opened a new log
I20260809 04:37:52.638386 8939 ts_tablet_manager.cc:1397] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 04:37:52.638763 8936 log.cc:824] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:52.640875 8936 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: No bootstrap required, opened a new log
I20260809 04:37:52.641074 8936 ts_tablet_manager.cc:1397] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:37:52.641475 8939 raft_consensus.cc:348] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.641697 8939 raft_consensus.cc:374] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:52.641757 8939 raft_consensus.cc:729] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7613783bd524ea1851327bc707711e8, State: Initialized, Role: FOLLOWER
I20260809 04:37:52.641700 8935 raft_consensus.cc:348] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.641952 8935 raft_consensus.cc:374] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:52.642024 8935 raft_consensus.cc:729] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cd6219ec254798858cbc9739ae5519, State: Initialized, Role: FOLLOWER
I20260809 04:37:52.642230 8939 consensus_queue.cc:260] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.642450 8935 consensus_queue.cc:260] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.643268 8939 ts_tablet_manager.cc:1428] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:37:52.643450 8935 ts_tablet_manager.cc:1428] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:37:52.643929 8461 heartbeater.cc:503] Master 127.19.92.62:41143 was elected leader, sending a full tablet report...
I20260809 04:37:52.643929 8687 heartbeater.cc:503] Master 127.19.92.62:41143 was elected leader, sending a full tablet report...
I20260809 04:37:52.644698 8936 raft_consensus.cc:348] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.644891 8936 raft_consensus.cc:374] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:52.644934 8936 raft_consensus.cc:729] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2735617de1401987f7556909143af3, State: Initialized, Role: FOLLOWER
I20260809 04:37:52.645344 8936 consensus_queue.cc:260] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.645970 8908 heartbeater.cc:503] Master 127.19.92.62:41143 was elected leader, sending a full tablet report...
I20260809 04:37:52.646255 8936 ts_tablet_manager.cc:1428] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 04:37:52.800346 8946 raft_consensus.cc:482] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:37:52.800638 8946 raft_consensus.cc:504] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.801913 8946 leader_election.cc:288] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce2735617de1401987f7556909143af3 (127.19.92.3:42307), e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
I20260809 04:37:52.806547 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.802483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:58543 (local address 127.19.92.2:33223)
0809 04:37:52.802608 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.802614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.802663 (+ 49us) server_negotiation.cc:407] Connection header received
0809 04:37:52.802871 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.802875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.802929 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.802980 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:52.803465 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.804146 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.805419 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.806011 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.806120 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.806189 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.806348 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.806422 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:37:52.806602 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.802179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33223 (local address 127.19.92.1:58543)
0809 04:37:52.802573 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.802588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.802744 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.803138 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.803141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.803179 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:52.803300 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.803307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.804248 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.804251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.805294 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.805305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.806139 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.806141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.806231 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.806259 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.806333 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.806404 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":54,"thread_start_us":113,"threads_started":1}
I20260809 04:37:52.806689 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.802725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:41365 (local address 127.19.92.3:42307)
0809 04:37:52.802851 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.802855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.803305 (+ 450us) server_negotiation.cc:407] Connection header received
0809 04:37:52.803418 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.803422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.803474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.803539 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:52.803800 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.804325 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.805526 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.806111 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.806197 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.806252 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.806474 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.806580 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:37:52.806928 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.802176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:42307 (local address 127.19.92.1:41365)
0809 04:37:52.803199 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.803214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.803317 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.803560 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.803563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.803578 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:52.803695 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.803703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.804436 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.804438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.805386 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.805396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.806226 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.806229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.806325 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.806344 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.806458 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.806517 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":705,"thread_start_us":141,"threads_started":1}
I20260809 04:37:52.807360 8614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf267bcbefb43269ef6aee8aca49122" candidate_uuid: "17cd6219ec254798858cbc9739ae5519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7613783bd524ea1851327bc707711e8" is_pre_election: true
I20260809 04:37:52.807495 8829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf267bcbefb43269ef6aee8aca49122" candidate_uuid: "17cd6219ec254798858cbc9739ae5519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2735617de1401987f7556909143af3" is_pre_election: true
I20260809 04:37:52.807693 8614 raft_consensus.cc:2454] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17cd6219ec254798858cbc9739ae5519 in term 0.
I20260809 04:37:52.807794 8829 raft_consensus.cc:2454] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17cd6219ec254798858cbc9739ae5519 in term 0.
I20260809 04:37:52.807811 8943 raft_consensus.cc:482] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:37:52.807902 8943 raft_consensus.cc:504] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.808224 8296 leader_election.cc:302] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519, ce2735617de1401987f7556909143af3, e7613783bd524ea1851327bc707711e8; no voters:
I20260809 04:37:52.808492 8946 raft_consensus.cc:2781] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:37:52.808579 8946 raft_consensus.cc:482] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:37:52.808617 8946 raft_consensus.cc:3037] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:52.809101 8943 leader_election.cc:288] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce2735617de1401987f7556909143af3 (127.19.92.3:42307), 17cd6219ec254798858cbc9739ae5519 (127.19.92.1:44411)
I20260809 04:37:52.810222 8946 raft_consensus.cc:504] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.810556 8946 leader_election.cc:288] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 1 election: Requested vote from peers ce2735617de1401987f7556909143af3 (127.19.92.3:42307), e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
I20260809 04:37:52.810887 8614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf267bcbefb43269ef6aee8aca49122" candidate_uuid: "17cd6219ec254798858cbc9739ae5519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7613783bd524ea1851327bc707711e8"
I20260809 04:37:52.811020 8614 raft_consensus.cc:3037] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:52.811089 8829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf267bcbefb43269ef6aee8aca49122" candidate_uuid: "17cd6219ec254798858cbc9739ae5519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2735617de1401987f7556909143af3"
I20260809 04:37:52.811216 8829 raft_consensus.cc:3037] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:52.812235 8614 raft_consensus.cc:2454] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17cd6219ec254798858cbc9739ae5519 in term 1.
I20260809 04:37:52.812476 8829 raft_consensus.cc:2454] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17cd6219ec254798858cbc9739ae5519 in term 1.
I20260809 04:37:52.812836 8294 leader_election.cc:302] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519, ce2735617de1401987f7556909143af3; no voters:
I20260809 04:37:52.813416 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.809296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:44411 (local address 127.19.92.2:37291)
0809 04:37:52.809383 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.809396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.809470 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.809679 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.809682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.809701 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:52.809803 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.809808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.810548 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.810552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.811678 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.811688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.813015 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.813018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.813114 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.813139 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.813225 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.813292 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 04:37:52.813731 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.809365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:37291 (local address 127.19.92.1:44411)
0809 04:37:52.809464 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.809468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.809491 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:37:52.809533 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.809536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.809574 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.809645 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:52.809903 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.810444 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.812408 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.812903 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.813019 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.813082 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.813563 (+ 481us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.813637 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:37:52.813868 8946 raft_consensus.cc:2781] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:37:52.814087 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.809365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:55145 (local address 127.19.92.3:42307)
0809 04:37:52.810720 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.810723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.810734 (+ 11us) server_negotiation.cc:407] Connection header received
0809 04:37:52.810794 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.810797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.810840 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.810906 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:52.811204 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.811840 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.813094 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.813559 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.813653 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.813705 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.813901 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.813971 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1279}
I20260809 04:37:52.814121 8946 raft_consensus.cc:686] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 1 LEADER]: Becoming Leader. State: Replica: 17cd6219ec254798858cbc9739ae5519, State: Running, Role: LEADER
I20260809 04:37:52.814198 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf267bcbefb43269ef6aee8aca49122" candidate_uuid: "e7613783bd524ea1851327bc707711e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cd6219ec254798858cbc9739ae5519" is_pre_election: true
I20260809 04:37:52.814391 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.809296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:42307 (local address 127.19.92.2:55145)
0809 04:37:52.809865 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.809879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:52.809965 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:52.810931 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:52.810934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:52.810953 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:52.811076 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.811083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.811977 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.811980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.812988 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:52.812999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.813667 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.813670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:52.813763 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.813785 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 04:37:52.814217 (+ 432us) client_negotiation.cc:240] Negotiation successful
0809 04:37:52.814290 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":231,"thread_start_us":98,"threads_started":1}
I20260809 04:37:52.814462 8946 consensus_queue.cc:237] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:52.816073 8829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf267bcbefb43269ef6aee8aca49122" candidate_uuid: "e7613783bd524ea1851327bc707711e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2735617de1401987f7556909143af3" is_pre_election: true
I20260809 04:37:52.816258 8829 raft_consensus.cc:2379] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7613783bd524ea1851327bc707711e8 in current term 1: Already voted for candidate 17cd6219ec254798858cbc9739ae5519 in this term.
I20260809 04:37:52.816691 8500 leader_election.cc:302] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7613783bd524ea1851327bc707711e8; no voters: 17cd6219ec254798858cbc9739ae5519, ce2735617de1401987f7556909143af3
I20260809 04:37:52.816948 8943 raft_consensus.cc:2726] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 04:37:52.817577 8910 tablet.cc:2369] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:37:52.817400 8165 catalog_manager.cc:5495] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17cd6219ec254798858cbc9739ae5519 (127.19.92.1). New cstate: current_term: 1 leader_uuid: "17cd6219ec254798858cbc9739ae5519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } health_report { overall_health: UNKNOWN } } }
W20260809 04:37:52.830916 8689 tablet.cc:2369] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:37:52.857646 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.853331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.19.92.1:44411)
0809 04:37:52.853457 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:52.853461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:52.853475 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:37:52.853632 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:52.853634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:52.853692 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:52.853777 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:37:52.853967 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.854659 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.855906 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:52.856220 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:52.856318 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:52.856647 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:37:52.857247 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:37:52.857364 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:52.857440 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 04:37:52.857505 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:37:52.863660 8829 raft_consensus.cc:1261] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 1 FOLLOWER]: Refusing update from remote peer 17cd6219ec254798858cbc9739ae5519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:37:52.863868 8614 raft_consensus.cc:1261] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Refusing update from remote peer 17cd6219ec254798858cbc9739ae5519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:37:52.864394 8955 consensus_queue.cc:1035] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:37:52.864639 8946 consensus_queue.cc:1035] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 04:37:52.866165 8462 tablet.cc:2369] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:37:52.880805 8174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:37:52.881008 8174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:37:52.883504 8174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Sending DeleteTablet for 3 replicas of tablet daf267bcbefb43269ef6aee8aca49122
I20260809 04:37:52.884241 8365 tablet_service.cc:1514] Processing DeleteTablet for tablet daf267bcbefb43269ef6aee8aca49122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:52 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:57170
I20260809 04:37:52.884406 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 8269
I20260809 04:37:52.884403 8798 tablet_service.cc:1514] Processing DeleteTablet for tablet daf267bcbefb43269ef6aee8aca49122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:52 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:53334
I20260809 04:37:52.885193 8972 tablet_replica.cc:330] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: stopping tablet replica
I20260809 04:37:52.885342 8579 tablet_service.cc:1514] Processing DeleteTablet for tablet daf267bcbefb43269ef6aee8aca49122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:52 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:40662
I20260809 04:37:52.885514 8972 raft_consensus.cc:2229] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:37:52.885840 8972 raft_consensus.cc:2258] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:37:52.886014 8973 tablet_replica.cc:330] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: stopping tablet replica
I20260809 04:37:52.886305 8973 raft_consensus.cc:2229] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:37:52.886605 8973 raft_consensus.cc:2258] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:37:52.891067 8973 ts_tablet_manager.cc:1905] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:37:52.893887 8973 ts_tablet_manager.cc:1918] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:37:52.893994 8973 log.cc:1196] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/daf267bcbefb43269ef6aee8aca49122
I20260809 04:37:52.894178 8972 ts_tablet_manager.cc:1905] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:37:52.894618 8973 ts_tablet_manager.cc:1939] T daf267bcbefb43269ef6aee8aca49122 P e7613783bd524ea1851327bc707711e8: Deleting consensus metadata
I20260809 04:37:52.895961 8146 catalog_manager.cc:4841] TS e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223): tablet daf267bcbefb43269ef6aee8aca49122 (table test-table [id=18031a2fdecc4278a6aa9e09703d0bb4]) successfully deleted
I20260809 04:37:52.896373 8972 ts_tablet_manager.cc:1918] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:37:52.896463 8972 log.cc:1196] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/daf267bcbefb43269ef6aee8aca49122
I20260809 04:37:52.896745 8972 ts_tablet_manager.cc:1939] T daf267bcbefb43269ef6aee8aca49122 P ce2735617de1401987f7556909143af3: Deleting consensus metadata
I20260809 04:37:52.897821 8145 catalog_manager.cc:4841] TS ce2735617de1401987f7556909143af3 (127.19.92.3:42307): tablet daf267bcbefb43269ef6aee8aca49122 (table test-table [id=18031a2fdecc4278a6aa9e09703d0bb4]) successfully deleted
W20260809 04:37:52.899472 8148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:44411 (error 108)
I20260809 04:37:52.900229 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 8469
I20260809 04:37:52.900480 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:52.900197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:44411 (local address 127.0.0.1:57196)
0809 04:37:52.900329 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:52.900397 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 04:37:52.905949 8148 catalog_manager.cc:4584] TS 17cd6219ec254798858cbc9739ae5519 (127.19.92.1:44411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet daf267bcbefb43269ef6aee8aca49122: Network error: Client connection negotiation failed: client connection to 127.19.92.1:44411: connect: Connection refused (error 111)
I20260809 04:37:52.913123 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 8695
I20260809 04:37:52.925612 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 8118
I20260809 04:37:52.938170 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41143
--webserver_interface=127.19.92.62
--webserver_port=40895
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:53.051254 8977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:53.051554 8977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:53.051614 8977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:53.055652 8977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:53.055711 8977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:53.055738 8977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:53.055760 8977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:53.055786 8977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:53.061062 8977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=40895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:53.062163 8977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:53.063194 8977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:53.078771 8986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:53.078770 8992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:53.078954 8977 server_base.cc:1034] running on GCE node
W20260809 04:37:53.078770 8988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:53.079898 8977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:53.081194 8977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:53.082372 8977 hybrid_clock.cc:648] HybridClock initialized: now 1786250273082351 us; error 39 us; skew 500 ppm
Aug 09 04:37:53 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:53.085263 8977 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:37:53.086197 8977 webserver.cc:460] Webserver started at http://127.19.92.62:40895/ using document root <none> and password file <none>
I20260809 04:37:53.086781 8977 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:53.086859 8977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:53.090881 8977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:37:53.093089 9003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:53.094364 8977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:37:53.094493 8977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "cf7cca8eed144dd2870a19f74e6c7797"
format_stamp: "Formatted at 2026-08-09 04:37:51 on dist-test-slave-rmhg"
server_key: "9f8975f5d9a6aae617a850e79dcd3525"
server_key_iv: "f5267fb9e33380acd3a7041b1f6b053f"
server_key_version: "encryptionkey@0"
I20260809 04:37:53.095090 8977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:53.110119 8977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:53.112921 8977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:53.113094 8977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:53.120440 8977 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41143
I20260809 04:37:53.120745 9082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41143 every 8 connection(s)
I20260809 04:37:53.121716 8977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:37:53.125139 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 8977
I20260809 04:37:53.125600 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:44411
--local_ip_for_outbound_sockets=127.19.92.1
--tserver_master_addrs=127.19.92.62:41143
--webserver_port=38569
--webserver_interface=127.19.92.1
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:37:53.126444 9083 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:37:53.128535 9083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Bootstrap starting.
I20260809 04:37:53.151755 9083 log.cc:824] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:53.163728 9083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:37:53.164199 9083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Bootstrap complete.
I20260809 04:37:53.169734 9083 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:53.170313 9083 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7cca8eed144dd2870a19f74e6c7797, State: Initialized, Role: FOLLOWER
I20260809 04:37:53.170960 9083 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:53.171139 9083 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:37:53.171221 9083 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:37:53.171325 9083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:37:53.174023 9083 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:53.174429 9083 leader_election.cc:302] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf7cca8eed144dd2870a19f74e6c7797; no voters:
I20260809 04:37:53.174798 9083 leader_election.cc:288] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 04:37:53.174940 9088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:37:53.175211 9088 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 2 LEADER]: Becoming Leader. State: Replica: cf7cca8eed144dd2870a19f74e6c7797, State: Running, Role: LEADER
I20260809 04:37:53.175555 9088 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:53.176144 9083 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:53.177067 9089 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf7cca8eed144dd2870a19f74e6c7797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } } }
I20260809 04:37:53.177219 9089 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:53.177045 9091 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf7cca8eed144dd2870a19f74e6c7797. Latest consensus state: current_term: 2 leader_uuid: "cf7cca8eed144dd2870a19f74e6c7797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } } }
I20260809 04:37:53.177316 9091 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:53.177942 9098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:37:53.182993 9098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:37:53.183652 9098 catalog_manager.cc:1259] Loaded cluster ID: dc3cede3202c4593af7b49b1743dee88
I20260809 04:37:53.183719 9098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:37:53.185096 9098 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 04:37:53.189702 9098 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 04:37:53.195748 9098 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:37:53.196597 9098 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Loaded TSK: 0
I20260809 04:37:53.197933 9098 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:37:53.263139 9086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:53.263370 9086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:53.263410 9086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:53.267379 9086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:53.267445 9086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:53.267503 9086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:37:53.272615 9086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:44411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=38569
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:53.273650 9086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:53.274626 9086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:53.285571 9119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:53.285707 9086 server_base.cc:1034] running on GCE node
W20260809 04:37:53.285872 9123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:53.285544 9117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:53.286350 9086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:53.287029 9086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:53.288201 9086 hybrid_clock.cc:648] HybridClock initialized: now 1786250273288173 us; error 42 us; skew 500 ppm
Aug 09 04:37:53 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:53.290931 9086 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:37:53.291930 9086 webserver.cc:460] Webserver started at http://127.19.92.1:38569/ using document root <none> and password file <none>
I20260809 04:37:53.292496 9086 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:53.292587 9086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:53.296607 9086 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:37:53.298933 9133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:53.300222 9086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:37:53.300326 9086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "17cd6219ec254798858cbc9739ae5519"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ad924b58b1ed082076ea91a93404b300"
server_key_iv: "491afa55c472e82878303972115f2dec"
server_key_version: "encryptionkey@0"
I20260809 04:37:53.300875 9086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:53.351996 9086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:53.355341 9086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:53.355527 9086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:53.356173 9086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:53.357353 9146 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 04:37:53.361779 9086 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 04:37:53.361871 9086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260809 04:37:53.361932 9086 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 04:37:53.363736 9086 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 04:37:53.363785 9086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 04:37:53.364089 9146 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Bootstrap starting.
I20260809 04:37:53.373313 9086 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:44411
I20260809 04:37:53.373406 9306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:44411 every 8 connection(s)
I20260809 04:37:53.374280 9086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:37:53.374819 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 9086
I20260809 04:37:53.375377 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:33223
--local_ip_for_outbound_sockets=127.19.92.2
--tserver_master_addrs=127.19.92.62:41143
--webserver_port=39655
--webserver_interface=127.19.92.2
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:37:53.383581 9146 log.cc:824] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:37:53 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:53.391289 9146 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:37:53.391701 9146 tablet_bootstrap.cc:492] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Bootstrap complete.
I20260809 04:37:53.392792 9146 ts_tablet_manager.cc:1397] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.004s
I20260809 04:37:53.394528 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.376818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:59375 (local address 127.19.92.62:41143)
0809 04:37:53.377264 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:53.377287 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:53.378047 (+ 760us) server_negotiation.cc:407] Connection header received
0809 04:37:53.379202 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:53.379216 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:53.379612 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:53.379979 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:53.381571 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.383079 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.384822 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.385554 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.385731 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.388811 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:53.388819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:53.388837 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:53.388873 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:53.390925 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.391512 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.391519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.391525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.391672 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.392054 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.392057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.392059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.392547 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:53.392675 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:53.393367 (+ 692us) server_negotiation.cc:299] Negotiation successful
0809 04:37:53.393670 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
I20260809 04:37:53.394551 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.376837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:59375)
0809 04:37:53.377886 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:53.377930 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:53.378768 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:53.380124 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:53.380137 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:53.380649 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:53.381362 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.381387 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.383265 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.383269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.384681 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.384695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.385724 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.385729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.385943 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.386645 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:53.386668 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:53.388585 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:53.391057 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:53.391062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:53.391074 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:53.391407 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:53.391785 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:53.391788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:53.391790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:53.391964 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:53.392658 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:53.392664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:53.393003 (+ 339us) client_negotiation.cc:769] Sending connection context
0809 04:37:53.393241 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 04:37:53.393659 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":147,"threads_started":1}
I20260809 04:37:53.395664 9146 raft_consensus.cc:348] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:53.396154 9146 raft_consensus.cc:729] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cd6219ec254798858cbc9739ae5519, State: Initialized, Role: FOLLOWER
I20260809 04:37:53.396608 9146 consensus_queue.cc:260] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } }
I20260809 04:37:53.397295 9308 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41143
I20260809 04:37:53.397547 9308 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:53.397608 9146 ts_tablet_manager.cc:1428] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 04:37:53.398164 9308 heartbeater.cc:511] Master 127.19.92.62:41143 requested a full tablet report, sending...
I20260809 04:37:53.400727 9026 ts_manager.cc:194] Registered new tserver with Master: 17cd6219ec254798858cbc9739ae5519 (127.19.92.1:44411)
I20260809 04:37:53.404304 9026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:59375
I20260809 04:37:53.407080 9308 heartbeater.cc:503] Master 127.19.92.62:41143 was elected leader, sending a full tablet report...
Aug 09 04:37:53 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:37:53.415526 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.403693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.19.92.1:44411)
0809 04:37:53.404350 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:53.404362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:53.404498 (+ 136us) server_negotiation.cc:407] Connection header received
0809 04:37:53.404581 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:53.404586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:53.404818 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:53.404952 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:53.407713 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.408424 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.409395 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.409726 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.409818 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.412020 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:53.412030 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:53.412039 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:53.412061 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:53.413601 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.414141 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.414147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.414149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.414238 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.414507 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.414510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.414512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.414776 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:53.414895 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:53.415236 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 04:37:53.415367 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":115,"threads_started":1}
I20260809 04:37:53.415551 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.403593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:44411 (local address 127.0.0.1:57200)
0809 04:37:53.404222 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:53.404271 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:53.404401 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:53.405088 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:53.405095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:53.407393 (+ 2298us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:53.407562 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.407572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.408538 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.408542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.409286 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.409297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.409834 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.409837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.409948 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.410465 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:53.410477 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:53.411879 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:53.413710 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:53.413719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:53.413724 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:53.414021 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:53.414326 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:53.414329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:53.414330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:53.414426 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:53.414889 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:53.414895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:53.415031 (+ 136us) client_negotiation.cc:769] Sending connection context
0809 04:37:53.415178 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 04:37:53.415376 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":119,"threads_started":1}
I20260809 04:37:53.416747 9216 tablet_service.cc:1514] Processing DeleteTablet for tablet daf267bcbefb43269ef6aee8aca49122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:52 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:57200
I20260809 04:37:53.417577 9323 tablet_replica.cc:330] stopping tablet replica
I20260809 04:37:53.417856 9323 raft_consensus.cc:2229] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:37:53.418093 9323 raft_consensus.cc:2258] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:37:53.419021 9323 ts_tablet_manager.cc:1905] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:37:53.422408 9323 ts_tablet_manager.cc:1918] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:37:53.422502 9323 log.cc:1196] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/daf267bcbefb43269ef6aee8aca49122
I20260809 04:37:53.422716 9323 ts_tablet_manager.cc:1939] T daf267bcbefb43269ef6aee8aca49122 P 17cd6219ec254798858cbc9739ae5519: Deleting consensus metadata
I20260809 04:37:53.424314 9007 catalog_manager.cc:4841] TS 17cd6219ec254798858cbc9739ae5519 (127.19.92.1:44411): tablet daf267bcbefb43269ef6aee8aca49122 (table test-table [id=18031a2fdecc4278a6aa9e09703d0bb4]) successfully deleted
W20260809 04:37:53.501751 9311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:53.502035 9311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:53.502097 9311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:53.506651 9311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:53.506719 9311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:53.506803 9311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:37:53.512214 9311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:33223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=39655
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:53.513376 9311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:53.514444 9311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:53.524519 9337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:53.524497 9339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:53.524505 9336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:53.524852 9311 server_base.cc:1034] running on GCE node
I20260809 04:37:53.525328 9311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:53.526010 9311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:53.527163 9311 hybrid_clock.cc:648] HybridClock initialized: now 1786250273527132 us; error 45 us; skew 500 ppm
Aug 09 04:37:53 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:53.529946 9311 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:37:53.530869 9311 webserver.cc:460] Webserver started at http://127.19.92.2:39655/ using document root <none> and password file <none>
I20260809 04:37:53.531458 9311 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:53.531545 9311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:53.535511 9311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:37:53.537690 9351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:53.538923 9311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:37:53.539049 9311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "e7613783bd524ea1851327bc707711e8"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ca44a4651a35dfd56eb557e1ed95da89"
server_key_iv: "fabe24e38cc7a23520b4378c2e4425d0"
server_key_version: "encryptionkey@0"
I20260809 04:37:53.539585 9311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:53.565379 9311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:53.568732 9311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:53.568919 9311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:53.569590 9311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:53.570462 9311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:53.570510 9311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:53.570551 9311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:53.570580 9311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:53.579344 9311 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:33223
I20260809 04:37:53.579366 9523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:33223 every 8 connection(s)
I20260809 04:37:53.580372 9311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:37:53.586706 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 9311
I20260809 04:37:53.587213 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:42307
--local_ip_for_outbound_sockets=127.19.92.3
--tserver_master_addrs=127.19.92.62:41143
--webserver_port=35143
--webserver_interface=127.19.92.3
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:37:53 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:53.595155 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.582237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:42613 (local address 127.19.92.62:41143)
0809 04:37:53.582375 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:53.582378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:53.583257 (+ 879us) server_negotiation.cc:407] Connection header received
0809 04:37:53.584571 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:53.584574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:53.584623 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:53.584697 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:53.585885 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.586426 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.588036 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.588275 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.588364 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.591221 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:53.591226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:53.591229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:53.591261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:53.592945 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.593522 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.593526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.593528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.593627 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.593937 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.593940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.593942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.594199 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:53.594305 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:53.594810 (+ 505us) server_negotiation.cc:299] Negotiation successful
0809 04:37:53.594972 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:37:53.595888 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.582628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:42613)
0809 04:37:53.583096 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:53.583140 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:53.584364 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:53.584852 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:53.584858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:53.585255 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:53.585753 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.585767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.586584 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.586588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.587882 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.587894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.588436 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.588440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.588645 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.589323 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:53.589347 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:53.591023 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:53.593067 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:53.593073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:53.593086 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:53.593441 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:53.593712 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:53.593715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:53.593716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:53.593849 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:53.594293 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:53.594299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:53.594557 (+ 258us) client_negotiation.cc:769] Sending connection context
0809 04:37:53.594714 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 04:37:53.594986 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":119,"threads_started":1}
I20260809 04:37:53.597090 9524 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41143
I20260809 04:37:53.597342 9524 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:53.597846 9524 heartbeater.cc:511] Master 127.19.92.62:41143 requested a full tablet report, sending...
I20260809 04:37:53.598955 9026 ts_manager.cc:194] Registered new tserver with Master: e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
I20260809 04:37:53.599565 9026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:42613
W20260809 04:37:53.706738 9532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:53.706987 9532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:53.707032 9532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:53.711001 9532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:53.711066 9532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:53.711126 9532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:37:53.716408 9532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:42307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=35143
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:53.717612 9532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:53.718752 9532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:53.728914 9544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:53.729091 9532 server_base.cc:1034] running on GCE node
W20260809 04:37:53.728967 9542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:53.729000 9541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:53.729646 9532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:53.730270 9532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:53.731428 9532 hybrid_clock.cc:648] HybridClock initialized: now 1786250273731399 us; error 46 us; skew 500 ppm
Aug 09 04:37:53 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:53.734210 9532 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:37:53.735127 9532 webserver.cc:460] Webserver started at http://127.19.92.3:35143/ using document root <none> and password file <none>
I20260809 04:37:53.735709 9532 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:53.735787 9532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:53.739918 9532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:37:53.742012 9556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:53.743299 9532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:37:53.743399 9532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "ce2735617de1401987f7556909143af3"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "bd6cee0ce9628662802031e4499c5def"
server_key_iv: "d10153caca78480f010ac539c3412a59"
server_key_version: "encryptionkey@0"
I20260809 04:37:53.743965 9532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:53.777483 9532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:53.780786 9532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:53.780980 9532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:53.781718 9532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:53.782629 9532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:53.782675 9532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:53.782716 9532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:53.782745 9532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:53.791725 9532 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:42307
I20260809 04:37:53.791782 9739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:42307 every 8 connection(s)
I20260809 04:37:53.792788 9532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:37:53.796304 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 9532
Aug 09 04:37:53 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250273, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:37:53.805610 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.796952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.19.92.62:41143)
0809 04:37:53.797220 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:53.797224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:53.797476 (+ 252us) server_negotiation.cc:407] Connection header received
0809 04:37:53.797606 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:53.797609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:53.797660 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:53.797749 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:53.798621 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.799294 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.800283 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.800634 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.800743 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.802195 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:53.802200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:53.802203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:53.802234 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:53.803785 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.804211 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.804213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.804215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.804295 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.804622 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.804625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.804627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.804845 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:53.805077 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:53.805271 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 04:37:53.805399 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260809 04:37:53.808123 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.794692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:37325 (local address 127.19.92.62:41143)
0809 04:37:53.794833 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:53.794837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:53.795739 (+ 902us) server_negotiation.cc:407] Connection header received
0809 04:37:53.797350 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:53.797353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:53.797410 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:53.797492 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:53.799003 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.799693 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.801193 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.801504 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.801621 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.804491 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:53.804495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:53.804497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:53.804520 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:53.805953 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.806544 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.806547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.806548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.806631 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:53.807014 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:53.807017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:53.807019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:53.807187 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:53.807275 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:53.807838 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 04:37:53.807977 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:37:53.809002 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.795089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:37325)
0809 04:37:53.795585 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:53.795631 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:53.797096 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:53.797713 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:53.797723 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:53.798160 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:53.798851 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.798873 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.799817 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.799820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.801044 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.801057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.801663 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.801666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.801823 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.802345 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:53.802363 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:53.804264 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:53.806067 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:53.806074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:53.806091 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:53.806428 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:53.806749 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:53.806752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:53.806754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:53.806911 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:53.807290 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:53.807297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:53.807601 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 04:37:53.807780 (+ 179us) client_negotiation.cc:240] Negotiation successful
0809 04:37:53.808132 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260809 04:37:53.810093 9741 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41143
I20260809 04:37:53.810340 9741 heartbeater.cc:464] Registering TS with master...
I20260809 04:37:53.810846 9741 heartbeater.cc:511] Master 127.19.92.62:41143 requested a full tablet report, sending...
I20260809 04:37:53.811957 9026 ts_manager.cc:194] Registered new tserver with Master: ce2735617de1401987f7556909143af3 (127.19.92.3:42307)
I20260809 04:37:53.812510 9026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:37325
I20260809 04:37:53.819234 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:37:53.823472 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.819882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.19.92.62:41143)
0809 04:37:53.820005 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:53.820008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:53.820225 (+ 217us) server_negotiation.cc:407] Connection header received
0809 04:37:53.820309 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:53.820311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:53.820409 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:53.820507 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:37:53.820753 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.821318 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.822191 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.822395 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.822484 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.822727 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:37:53.823138 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:37:53.823224 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:53.823298 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 04:37:53.823343 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:37:53.824105 9026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:37:53.827422 9026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:37:53.835196 9216 tablet_service.cc:1467] Processing CreateTablet for tablet 09fa5219cbcc43fc995645583060658b (DEFAULT_TABLE table=test-table [id=a408d96cfe6e43f9ab61dcbb13946998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:53.835937 9216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09fa5219cbcc43fc995645583060658b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:53.838681 9146 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Bootstrap starting.
I20260809 04:37:53.839700 9146 tablet_bootstrap.cc:654] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:53.840301 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.834556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33223 (local address 127.0.0.1:40678)
0809 04:37:53.835126 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:53.835141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:53.835242 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:53.835732 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:53.835735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:53.835754 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:53.835842 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.835849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.836920 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.836923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.838474 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.838486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.839882 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.839886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.839984 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.840025 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 04:37:53.840079 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 04:37:53.840148 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":127,"thread_start_us":79,"threads_started":1}
I20260809 04:37:53.840699 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.834099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:42307 (local address 127.0.0.1:53350)
0809 04:37:53.834215 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:53.834231 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:53.834312 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:53.835398 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:53.835401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:53.835419 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:53.835537 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.835553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.836969 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.836972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.838473 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:53.838486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.840270 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.840273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:53.840369 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.840393 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:37:53.840497 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 04:37:53.840586 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":124}
I20260809 04:37:53.840791 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.834730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.19.92.2:33223)
0809 04:37:53.835067 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:53.835077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:53.835229 (+ 152us) server_negotiation.cc:407] Connection header received
0809 04:37:53.835352 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:53.835358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:53.835564 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:53.835705 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:53.835955 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.836788 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.838631 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.839507 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.839617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.840487 (+ 870us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:53.840544 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 04:37:53.840651 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
I20260809 04:37:53.841279 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:53.834325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.19.92.3:42307)
0809 04:37:53.834838 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:53.834848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:53.834869 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:37:53.834965 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:53.834971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:53.835202 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:53.835409 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:53.835686 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.836852 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.839026 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:53.840122 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:53.840246 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:53.840404 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:53.840991 (+ 587us) server_negotiation.cc:299] Negotiation successful
0809 04:37:53.841100 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":157,"threads_started":1}
I20260809 04:37:53.841624 9146 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: No bootstrap required, opened a new log
I20260809 04:37:53.841727 9146 ts_tablet_manager.cc:1397] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:37:53.842237 9146 raft_consensus.cc:348] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:53.842341 9146 raft_consensus.cc:374] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:53.842367 9146 raft_consensus.cc:729] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cd6219ec254798858cbc9739ae5519, State: Initialized, Role: FOLLOWER
I20260809 04:37:53.842478 9146 consensus_queue.cc:260] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:53.842700 9146 ts_tablet_manager.cc:1428] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 04:37:53.844204 9412 tablet_service.cc:1467] Processing CreateTablet for tablet 09fa5219cbcc43fc995645583060658b (DEFAULT_TABLE table=test-table [id=a408d96cfe6e43f9ab61dcbb13946998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:53.844204 9629 tablet_service.cc:1467] Processing CreateTablet for tablet 09fa5219cbcc43fc995645583060658b (DEFAULT_TABLE table=test-table [id=a408d96cfe6e43f9ab61dcbb13946998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:37:53.845438 9412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09fa5219cbcc43fc995645583060658b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:53.845456 9629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09fa5219cbcc43fc995645583060658b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:37:53.853981 9758 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Bootstrap starting.
I20260809 04:37:53.854645 9757 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Bootstrap starting.
I20260809 04:37:53.856738 9758 tablet_bootstrap.cc:654] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:53.857256 9757 tablet_bootstrap.cc:654] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Neither blocks nor log segments found. Creating new log.
I20260809 04:37:53.858083 9758 log.cc:824] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:53.858083 9757 log.cc:824] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:53.860220 9758 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: No bootstrap required, opened a new log
I20260809 04:37:53.860428 9758 ts_tablet_manager.cc:1397] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 04:37:53.862627 9757 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: No bootstrap required, opened a new log
I20260809 04:37:53.862828 9757 ts_tablet_manager.cc:1397] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260809 04:37:53.864171 9758 raft_consensus.cc:348] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:53.864482 9758 raft_consensus.cc:374] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:53.864547 9758 raft_consensus.cc:729] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2735617de1401987f7556909143af3, State: Initialized, Role: FOLLOWER
I20260809 04:37:53.865034 9758 consensus_queue.cc:260] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:53.866035 9741 heartbeater.cc:503] Master 127.19.92.62:41143 was elected leader, sending a full tablet report...
I20260809 04:37:53.866117 9758 ts_tablet_manager.cc:1428] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260809 04:37:53.866585 9757 raft_consensus.cc:348] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:53.866773 9757 raft_consensus.cc:374] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:37:53.866806 9757 raft_consensus.cc:729] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7613783bd524ea1851327bc707711e8, State: Initialized, Role: FOLLOWER
I20260809 04:37:53.867209 9757 consensus_queue.cc:260] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:53.867954 9524 heartbeater.cc:503] Master 127.19.92.62:41143 was elected leader, sending a full tablet report...
I20260809 04:37:53.868088 9757 ts_tablet_manager.cc:1428] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260809 04:37:53.875507 9309 tablet.cc:2369] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:37:54.043747 9743 tablet.cc:2369] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:37:54.081589 9529 tablet.cc:2369] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:37:54.214231 9762 raft_consensus.cc:482] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:37:54.214461 9762 raft_consensus.cc:504] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:54.215643 9762 leader_election.cc:288] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223), 17cd6219ec254798858cbc9739ae5519 (127.19.92.1:44411)
I20260809 04:37:54.219627 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.216045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:42219 (local address 127.19.92.2:33223)
0809 04:37:54.216170 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:54.216175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:54.216547 (+ 372us) server_negotiation.cc:407] Connection header received
0809 04:37:54.216635 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:54.216638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:54.216684 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:54.216758 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:54.217030 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.217561 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.218686 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.219078 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.219155 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:54.219222 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:54.219422 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 04:37:54.219495 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:37:54.219627 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.215949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:44411 (local address 127.19.92.3:40699)
0809 04:37:54.216083 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:54.216101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:54.216216 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:54.216716 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:54.216719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:54.216736 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:54.216872 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:54.216878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.217671 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.217675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:54.218529 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:54.218539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.219222 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.219225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:54.219324 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:54.219341 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 04:37:54.219396 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 04:37:54.219490 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":24}
I20260809 04:37:54.219642 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.216220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:40699 (local address 127.19.92.1:44411)
0809 04:37:54.216471 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:54.216474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:54.216489 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:37:54.216539 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:54.216541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:54.216597 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:54.216694 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:37:54.216984 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.217563 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.218655 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.219106 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.219209 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:54.219389 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:54.219427 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 04:37:54.219497 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
I20260809 04:37:54.219652 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.215948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33223 (local address 127.19.92.3:42219)
0809 04:37:54.216451 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:54.216461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:54.216550 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:54.216797 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:54.216800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:54.216820 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:37:54.216917 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:54.216922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.217671 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.217675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:54.218561 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:54.218572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.219201 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.219204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:54.219295 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:54.219319 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:37:54.219393 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:37:54.219483 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":109,"thread_start_us":154,"threads_started":1}
I20260809 04:37:54.220363 9451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fa5219cbcc43fc995645583060658b" candidate_uuid: "ce2735617de1401987f7556909143af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7613783bd524ea1851327bc707711e8" is_pre_election: true
I20260809 04:37:54.220376 9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fa5219cbcc43fc995645583060658b" candidate_uuid: "ce2735617de1401987f7556909143af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cd6219ec254798858cbc9739ae5519" is_pre_election: true
I20260809 04:37:54.220693 9242 raft_consensus.cc:2454] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce2735617de1401987f7556909143af3 in term 0.
I20260809 04:37:54.220693 9451 raft_consensus.cc:2454] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce2735617de1401987f7556909143af3 in term 0.
I20260809 04:37:54.221243 9563 leader_election.cc:302] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519, ce2735617de1401987f7556909143af3; no voters:
I20260809 04:37:54.221553 9762 raft_consensus.cc:2781] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:37:54.221649 9762 raft_consensus.cc:482] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:37:54.221696 9762 raft_consensus.cc:3037] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:54.222748 9762 raft_consensus.cc:504] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:54.223110 9762 leader_election.cc:288] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [CANDIDATE]: Term 1 election: Requested vote from peers e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223), 17cd6219ec254798858cbc9739ae5519 (127.19.92.1:44411)
I20260809 04:37:54.223577 9242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fa5219cbcc43fc995645583060658b" candidate_uuid: "ce2735617de1401987f7556909143af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cd6219ec254798858cbc9739ae5519"
I20260809 04:37:54.223718 9242 raft_consensus.cc:3037] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:54.224750 9242 raft_consensus.cc:2454] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce2735617de1401987f7556909143af3 in term 1.
I20260809 04:37:54.225029 9451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fa5219cbcc43fc995645583060658b" candidate_uuid: "ce2735617de1401987f7556909143af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7613783bd524ea1851327bc707711e8"
I20260809 04:37:54.225081 9563 leader_election.cc:302] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519, ce2735617de1401987f7556909143af3; no voters:
I20260809 04:37:54.225176 9451 raft_consensus.cc:3037] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:37:54.225322 9762 raft_consensus.cc:2781] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:37:54.225602 9762 raft_consensus.cc:686] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 1 LEADER]: Becoming Leader. State: Replica: ce2735617de1401987f7556909143af3, State: Running, Role: LEADER
I20260809 04:37:54.226241 9762 consensus_queue.cc:237] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:54.226560 9451 raft_consensus.cc:2454] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce2735617de1401987f7556909143af3 in term 1.
I20260809 04:37:54.229679 9025 catalog_manager.cc:5495] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce2735617de1401987f7556909143af3 (127.19.92.3). New cstate: current_term: 1 leader_uuid: "ce2735617de1401987f7556909143af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } health_report { overall_health: UNKNOWN } } }
I20260809 04:37:54.301594 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.298251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.19.92.3:42307)
0809 04:37:54.298395 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:54.298399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:54.298412 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:37:54.298530 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:54.298533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:54.298595 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:54.298677 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:37:54.298885 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.299383 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.300217 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:54.300430 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:54.300526 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:54.300769 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:37:54.301215 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:37:54.301302 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:54.301367 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 04:37:54.301419 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:37:54.307720 9451 raft_consensus.cc:1261] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Refusing update from remote peer ce2735617de1401987f7556909143af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:37:54.307714 9242 raft_consensus.cc:1261] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Refusing update from remote peer ce2735617de1401987f7556909143af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:37:54.308440 9762 consensus_queue.cc:1035] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:37:54.308641 9769 consensus_queue.cc:1035] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:37:54.313433 9772 mvcc.cc:205] Tried to move back new op lower bound from 7316481123556544512 to 7316481123234021376. Current Snapshot: MvccSnapshot[applied={T|T < 7316481123556544512}]
I20260809 04:37:54.324951 9026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:37:54.325198 9026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:37:54.327744 9026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Sending DeleteTablet for 3 replicas of tablet 09fa5219cbcc43fc995645583060658b
I20260809 04:37:54.328444 9629 tablet_service.cc:1514] Processing DeleteTablet for tablet 09fa5219cbcc43fc995645583060658b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:54 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:53350
I20260809 04:37:54.328543 9412 tablet_service.cc:1514] Processing DeleteTablet for tablet 09fa5219cbcc43fc995645583060658b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:54 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:40678
I20260809 04:37:54.328874 9216 tablet_service.cc:1514] Processing DeleteTablet for tablet 09fa5219cbcc43fc995645583060658b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:37:54 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:57200
I20260809 04:37:54.328902 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 9086
I20260809 04:37:54.329012 9782 tablet_replica.cc:330] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: stopping tablet replica
I20260809 04:37:54.329365 9782 raft_consensus.cc:2229] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:37:54.329773 9782 raft_consensus.cc:2258] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:37:54.336947 9785 tablet_replica.cc:330] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: stopping tablet replica
I20260809 04:37:54.337513 9785 raft_consensus.cc:2229] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:37:54.338172 9785 raft_consensus.cc:2258] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:37:54.341307 9782 ts_tablet_manager.cc:1905] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:37:54.342680 9785 ts_tablet_manager.cc:1905] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:37:54.344264 9782 ts_tablet_manager.cc:1918] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:37:54.344364 9782 log.cc:1196] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/09fa5219cbcc43fc995645583060658b
I20260809 04:37:54.344712 9782 ts_tablet_manager.cc:1939] T 09fa5219cbcc43fc995645583060658b P e7613783bd524ea1851327bc707711e8: Deleting consensus metadata
I20260809 04:37:54.345678 9785 ts_tablet_manager.cc:1918] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:37:54.345770 9785 log.cc:1196] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/09fa5219cbcc43fc995645583060658b
I20260809 04:37:54.346065 9785 ts_tablet_manager.cc:1939] T 09fa5219cbcc43fc995645583060658b P ce2735617de1401987f7556909143af3: Deleting consensus metadata
I20260809 04:37:54.346271 9005 catalog_manager.cc:4841] TS e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223): tablet 09fa5219cbcc43fc995645583060658b (table test-table [id=a408d96cfe6e43f9ab61dcbb13946998]) successfully deleted
I20260809 04:37:54.347225 9004 catalog_manager.cc:4841] TS ce2735617de1401987f7556909143af3 (127.19.92.3:42307): tablet 09fa5219cbcc43fc995645583060658b (table test-table [id=a408d96cfe6e43f9ab61dcbb13946998]) successfully deleted
W20260809 04:37:54.347261 9007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:44411 (error 108)
I20260809 04:37:54.347879 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 9311
I20260809 04:37:54.351437 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.351043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:44411 (local address 127.0.0.1:57208)
0809 04:37:54.351270 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:54.351341 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":60,"threads_started":1}
W20260809 04:37:54.351691 9007 catalog_manager.cc:4584] TS 17cd6219ec254798858cbc9739ae5519 (127.19.92.1:44411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09fa5219cbcc43fc995645583060658b: Network error: Client connection negotiation failed: client connection to 127.19.92.1:44411: connect: Connection refused (error 111)
I20260809 04:37:54.358563 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 9532
I20260809 04:37:54.376178 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 8977
I20260809 04:37:54.385866 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41143
--webserver_interface=127.19.92.62
--webserver_port=40895
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:37:54.498724 9790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:54.499003 9790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:54.499060 9790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:54.503067 9790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:37:54.503122 9790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:54.503149 9790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:54.503172 9790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:37:54.503201 9790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:37:54.508383 9790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=40895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:54.509433 9790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:54.510421 9790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:54.519013 9796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:54.519115 9790 server_base.cc:1034] running on GCE node
W20260809 04:37:54.519009 9800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:54.519013 9797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:54.519840 9790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:54.521091 9790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:54.522274 9790 hybrid_clock.cc:648] HybridClock initialized: now 1786250274522249 us; error 43 us; skew 500 ppm
Aug 09 04:37:54 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:54.525033 9790 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:37:54.525981 9790 webserver.cc:460] Webserver started at http://127.19.92.62:40895/ using document root <none> and password file <none>
I20260809 04:37:54.526538 9790 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:54.526605 9790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:54.530470 9790 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:37:54.532843 9814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:54.534142 9790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:37:54.534262 9790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "cf7cca8eed144dd2870a19f74e6c7797"
format_stamp: "Formatted at 2026-08-09 04:37:51 on dist-test-slave-rmhg"
server_key: "9f8975f5d9a6aae617a850e79dcd3525"
server_key_iv: "f5267fb9e33380acd3a7041b1f6b053f"
server_key_version: "encryptionkey@0"
I20260809 04:37:54.534843 9790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:54.547497 9790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:54.550437 9790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:54.550604 9790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:54.557775 9790 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41143
I20260809 04:37:54.558085 9891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41143 every 8 connection(s)
I20260809 04:37:54.558938 9790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:37:54.562893 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 9790
I20260809 04:37:54.563318 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:44411
--local_ip_for_outbound_sockets=127.19.92.1
--tserver_master_addrs=127.19.92.62:41143
--webserver_port=38569
--webserver_interface=127.19.92.1
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:37:54.563603 9893 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:37:54.565440 9893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Bootstrap starting.
I20260809 04:37:54.583304 9893 log.cc:824] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Log is configured to *not* fsync() on all Append() calls
I20260809 04:37:54.594907 9893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:37:54.595276 9893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797: Bootstrap complete.
I20260809 04:37:54.599234 9893 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:54.599669 9893 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7cca8eed144dd2870a19f74e6c7797, State: Initialized, Role: FOLLOWER
I20260809 04:37:54.600210 9893 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:54.600345 9893 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:37:54.600419 9893 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:37:54.600522 9893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 2 FOLLOWER]: Advancing to term 3
I20260809 04:37:54.602607 9893 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:54.602928 9893 leader_election.cc:302] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf7cca8eed144dd2870a19f74e6c7797; no voters:
I20260809 04:37:54.603245 9893 leader_election.cc:288] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 04:37:54.603348 9899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 04:37:54.603554 9899 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [term 3 LEADER]: Becoming Leader. State: Replica: cf7cca8eed144dd2870a19f74e6c7797, State: Running, Role: LEADER
I20260809 04:37:54.603860 9899 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } }
I20260809 04:37:54.604559 9893 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:37:54.604866 9899 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf7cca8eed144dd2870a19f74e6c7797. Latest consensus state: current_term: 3 leader_uuid: "cf7cca8eed144dd2870a19f74e6c7797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } } }
I20260809 04:37:54.605087 9899 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:54.606300 9900 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cf7cca8eed144dd2870a19f74e6c7797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf7cca8eed144dd2870a19f74e6c7797" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41143 } } }
I20260809 04:37:54.606405 9904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:37:54.606494 9900 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cca8eed144dd2870a19f74e6c7797 [sys.catalog]: This master's current role is: LEADER
I20260809 04:37:54.611459 9904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:37:54.612102 9904 catalog_manager.cc:1259] Loaded cluster ID: dc3cede3202c4593af7b49b1743dee88
I20260809 04:37:54.612166 9904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 04:37:54.613323 9904 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc71c0699ad google::LogMessage::Fail() at ??:0
@ 0x7fc71c06c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc71c0694de google::LogMessage::Flush() at ??:0
@ 0x7fc71c06aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc7269a8fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc7269a9d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc7269a6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc7269ea640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc71e2bb3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc71e2bbc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc71e2bd3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc71e2ac512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc71f8f06db start_thread at ??:0
@ 0x7fc71ab6761f clone at ??:0
W20260809 04:37:54.703903 9896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:54.704195 9896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:54.704239 9896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:54.708339 9896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:54.708398 9896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:54.708460 9896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:37:54.713738 9896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:44411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=38569
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:54.714792 9896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:54.715798 9896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:54.725696 9959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:54.725696 9957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:54.725696 9956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:54.726181 9896 server_base.cc:1034] running on GCE node
I20260809 04:37:54.726608 9896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:54.727262 9896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:54.728436 9896 hybrid_clock.cc:648] HybridClock initialized: now 1786250274728396 us; error 53 us; skew 500 ppm
Aug 09 04:37:54 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:54.731280 9896 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:37:54.732254 9896 webserver.cc:460] Webserver started at http://127.19.92.1:38569/ using document root <none> and password file <none>
I20260809 04:37:54.732813 9896 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:54.732873 9896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:54.736764 9896 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260809 04:37:54.738891 9968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:54.740095 9896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:37:54.740195 9896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "17cd6219ec254798858cbc9739ae5519"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ad924b58b1ed082076ea91a93404b300"
server_key_iv: "491afa55c472e82878303972115f2dec"
server_key_version: "encryptionkey@0"
I20260809 04:37:54.740729 9896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:54.772477 9896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:54.775827 9896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:54.776021 9896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:54.776652 9896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:54.777810 9977 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 04:37:54.783536 9896 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 04:37:54.783614 9896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260809 04:37:54.783684 9896 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 04:37:54.785518 9896 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 04:37:54.785570 9896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260809 04:37:54.785934 9977 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Bootstrap starting.
I20260809 04:37:54.795714 9896 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:44411
I20260809 04:37:54.795771 10155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:44411 every 8 connection(s)
I20260809 04:37:54.796823 9896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:37:54.801817 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 9896
I20260809 04:37:54.802245 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:33223
--local_ip_for_outbound_sockets=127.19.92.2
--tserver_master_addrs=127.19.92.62:41143
--webserver_port=39655
--webserver_interface=127.19.92.2
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:37:54.808409 9977 log.cc:824] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Log is configured to *not* fsync() on all Append() calls
W20260809 04:37:54.817360 10159 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:37:54.798962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:47783)
0809 04:37:54.799524 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:54.799588 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:54.800461 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:54.816497 (+ 16036us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
W20260809 04:37:54.817830 10156 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:41143 (0 consecutive failures): Network error: Failed to ping master at 127.19.92.62:41143: Client connection negotiation failed: client connection to 127.19.92.62:41143: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 04:37:54.818624 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.818270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:37655)
0809 04:37:54.818500 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:54.818565 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":109}
I20260809 04:37:54.818679 9977 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:37:54.819082 9977 tablet_bootstrap.cc:492] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Bootstrap complete.
I20260809 04:37:54.819221 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.819005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:51635)
0809 04:37:54.819110 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:54.819160 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 04:37:54.819375 10156 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 04:37:54.820401 9977 ts_tablet_manager.cc:1397] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.005s
I20260809 04:37:54.823608 9977 raft_consensus.cc:348] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:54.824167 9977 raft_consensus.cc:729] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cd6219ec254798858cbc9739ae5519, State: Initialized, Role: FOLLOWER
I20260809 04:37:54.824749 9977 consensus_queue.cc:260] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:54.825848 9977 ts_tablet_manager.cc:1428] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260809 04:37:54.825858 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.825641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:51507)
0809 04:37:54.825744 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:54.825798 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 04:37:54.826538 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:54.826307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:38765)
0809 04:37:54.826413 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:54.826478 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 04:37:54.927057 10160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:54.927342 10160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:54.927386 10160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:54.931404 10160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:54.931463 10160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:54.931521 10160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:37:54.936832 10160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:33223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=39655
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:54.937923 10160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:54.938921 10160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:54.948648 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:54.948650 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:54.948648 10174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:54.948915 10160 server_base.cc:1034] running on GCE node
I20260809 04:37:54.949378 10160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:54.950047 10160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:54.951211 10160 hybrid_clock.cc:648] HybridClock initialized: now 1786250274951192 us; error 30 us; skew 500 ppm
Aug 09 04:37:54 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:54.953967 10160 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:37:54.954808 10160 webserver.cc:460] Webserver started at http://127.19.92.2:39655/ using document root <none> and password file <none>
I20260809 04:37:54.955343 10160 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:54.955416 10160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:54.959297 10160 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:37:54.961576 10186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:54.962896 10160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:37:54.963018 10160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "e7613783bd524ea1851327bc707711e8"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "ca44a4651a35dfd56eb557e1ed95da89"
server_key_iv: "fabe24e38cc7a23520b4378c2e4425d0"
server_key_version: "encryptionkey@0"
I20260809 04:37:54.963604 10160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:54.994493 10160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:54.997624 10160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:54.997820 10160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:54.998607 10160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:54.999655 10160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:54.999708 10160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:54.999765 10160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:54.999794 10160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:55.008914 10160 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:33223
I20260809 04:37:55.008945 10344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:33223 every 8 connection(s)
I20260809 04:37:55.010058 10160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:37:55.012475 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 10160
I20260809 04:37:55.012877 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:42307
--local_ip_for_outbound_sockets=127.19.92.3
--tserver_master_addrs=127.19.92.62:41143
--webserver_port=35143
--webserver_interface=127.19.92.3
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:37:55.014178 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:55.012346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:56497)
0809 04:37:55.012884 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:55.013432 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":120,"threads_started":1}
W20260809 04:37:55.014717 10345 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:41143 (0 consecutive failures): Network error: Failed to ping master at 127.19.92.62:41143: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
I20260809 04:37:55.015359 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:55.015126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:40013)
0809 04:37:55.015241 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:55.015295 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 04:37:55.015964 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:55.015744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:43983)
0809 04:37:55.015850 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:55.015897 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:37:55.016153 10345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:37:55.128667 10348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:37:55.128969 10348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:37:55.129022 10348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:37:55.133065 10348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:37:55.133141 10348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:37:55.133247 10348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:37:55.138903 10348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:42307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=35143
--tserver_master_addrs=127.19.92.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:37:55.140045 10348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:37:55.141068 10348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:37:55.151230 10356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:37:55.151237 10354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:55.151286 10348 server_base.cc:1034] running on GCE node
W20260809 04:37:55.151237 10358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:37:55.151878 10348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:37:55.152515 10348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:37:55.153702 10348 hybrid_clock.cc:648] HybridClock initialized: now 1786250275153681 us; error 30 us; skew 500 ppm
Aug 09 04:37:55 dist-test-slave-rmhg krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:37:55.156548 10348 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:37:55.157524 10348 webserver.cc:460] Webserver started at http://127.19.92.3:35143/ using document root <none> and password file <none>
I20260809 04:37:55.158087 10348 fs_manager.cc:359] Metadata directory not provided
I20260809 04:37:55.158149 10348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:37:55.162454 10348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:37:55.164649 10368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:55.166059 10348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:37:55.166178 10348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "ce2735617de1401987f7556909143af3"
format_stamp: "Formatted at 2026-08-09 04:37:52 on dist-test-slave-rmhg"
server_key: "bd6cee0ce9628662802031e4499c5def"
server_key_iv: "d10153caca78480f010ac539c3412a59"
server_key_version: "encryptionkey@0"
I20260809 04:37:55.166769 10348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:37:55.191767 10348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:37:55.194913 10348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:37:55.195094 10348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:37:55.195672 10348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:37:55.196523 10348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:37:55.196571 10348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:55.196612 10348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:37:55.196632 10348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:37:55.205832 10348 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:42307
I20260809 04:37:55.205858 10553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:42307 every 8 connection(s)
I20260809 04:37:55.206894 10348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:37:55.210876 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:55.209145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:34647)
0809 04:37:55.209695 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:55.210123 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260809 04:37:55.210894 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 10348
W20260809 04:37:55.211411 10556 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:41143 (0 consecutive failures): Network error: Failed to ping master at 127.19.92.62:41143: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
I20260809 04:37:55.212031 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:55.211802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:50565)
0809 04:37:55.211918 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:55.211977 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 04:37:55.212579 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:55.212398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:60867)
0809 04:37:55.212487 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:55.212528 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 04:37:55.212745 10556 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 04:37:55.827796 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:55.827330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:58587)
0809 04:37:55.827616 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:55.827678 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":97,"threads_started":1}
I20260809 04:37:56.017087 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:56.016692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:55471)
0809 04:37:56.016920 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:56.016977 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260809 04:37:56.183777 10567 raft_consensus.cc:482] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:37:56.183988 10567 raft_consensus.cc:504] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:56.185140 10567 leader_election.cc:288] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2735617de1401987f7556909143af3 (127.19.92.3:42307), e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
Aug 09 04:37:56 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.2@KRBTEST.COM
Aug 09 04:37:56 dist-test-slave-rmhg krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.3@KRBTEST.COM
I20260809 04:37:56.198741 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:56.185391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33223 (local address 127.19.92.1:33269)
0809 04:37:56.185940 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:56.185952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:56.186046 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:56.187329 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:56.187335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:56.187857 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:56.188320 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:56.188336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:56.189624 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:56.189627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:56.190597 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:56.190609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:56.191483 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:56.191486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:56.191643 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:56.192311 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:56.192330 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:56.193963 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:56.196202 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:56.196209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:56.196224 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:56.196569 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:56.196913 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:56.196916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:56.196918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:56.197090 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:56.197737 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:56.197743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:56.197998 (+ 255us) client_negotiation.cc:769] Sending connection context
0809 04:37:56.198169 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 04:37:56.198555 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":298,"spinlock_wait_cycles":45056,"thread_start_us":80,"threads_started":1}
I20260809 04:37:56.198755 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:56.185532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:60885 (local address 127.19.92.3:42307)
0809 04:37:56.185896 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:56.185910 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:56.185953 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:37:56.186603 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:56.186608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:56.186966 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:56.187224 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:56.188491 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:56.189505 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:56.190697 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:56.191309 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:56.191518 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:56.194559 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:56.194564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:56.194576 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:56.194605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:56.196122 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:56.196685 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:56.196692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:56.196697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:56.196818 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:56.197189 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:56.197193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:56.197194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:56.197621 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:56.197743 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:56.198299 (+ 556us) server_negotiation.cc:299] Negotiation successful
0809 04:37:56.198575 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260809 04:37:56.198750 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:56.185559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:33269 (local address 127.19.92.2:33223)
0809 04:37:56.186003 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0809 04:37:56.186027 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 04:37:56.186069 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:37:56.186716 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:37:56.186722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:37:56.187030 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:37:56.187300 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:37:56.188483 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:56.189477 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:56.190733 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:37:56.191325 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:37:56.191492 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:56.194214 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:37:56.194221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:37:56.194240 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:37:56.194277 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:37:56.196064 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:56.196686 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:56.196692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:56.196697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:56.196819 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:37:56.197199 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:37:56.197202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:37:56.197204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:37:56.197619 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:37:56.197743 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 04:37:56.198299 (+ 556us) server_negotiation.cc:299] Negotiation successful
0809 04:37:56.198554 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":115,"threads_started":1}
I20260809 04:37:56.198772 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:56.185391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:42307 (local address 127.19.92.1:60885)
0809 04:37:56.185612 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:56.185626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:37:56.185729 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:37:56.187328 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:37:56.187334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:37:56.187856 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:37:56.188322 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:56.188335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:56.189630 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:56.189632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:56.190590 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:37:56.190599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:37:56.191426 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:37:56.191430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:37:56.191641 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:37:56.192309 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:37:56.192334 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:37:56.194309 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:37:56.196239 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:56.196241 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:56.196243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:56.196572 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:56.196912 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:37:56.196915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:37:56.196918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:37:56.197078 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:37:56.197737 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:37:56.197743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:37:56.197999 (+ 256us) client_negotiation.cc:769] Sending connection context
0809 04:37:56.198169 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 04:37:56.198632 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":271,"spinlock_wait_cycles":19584}
W20260809 04:37:56.201574 9971 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
W20260809 04:37:56.201794 9969 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2735617de1401987f7556909143af3 (127.19.92.3:42307): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
I20260809 04:37:56.201889 9969 leader_election.cc:302] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519; no voters: ce2735617de1401987f7556909143af3, e7613783bd524ea1851327bc707711e8
I20260809 04:37:56.202171 10567 raft_consensus.cc:2726] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:37:56.213635 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:56.213252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:58327)
0809 04:37:56.213471 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:56.213547 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260809 04:37:56.829092 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:56.828645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:53867)
0809 04:37:56.828930 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:56.828989 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260809 04:37:57.018141 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:57.017775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:50067)
0809 04:37:57.017997 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:57.018049 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260809 04:37:57.214704 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:57.214289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:50211)
0809 04:37:57.214534 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:57.214590 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260809 04:37:57.739759 10579 raft_consensus.cc:482] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:37:57.739894 10579 raft_consensus.cc:504] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:37:57.740304 10579 leader_election.cc:288] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2735617de1401987f7556909143af3 (127.19.92.3:42307), e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
W20260809 04:37:57.741074 9969 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2735617de1401987f7556909143af3 (127.19.92.3:42307): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
W20260809 04:37:57.741215 9971 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
I20260809 04:37:57.741309 9971 leader_election.cc:302] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519; no voters: ce2735617de1401987f7556909143af3, e7613783bd524ea1851327bc707711e8
I20260809 04:37:57.741456 10579 raft_consensus.cc:2726] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:37:57.830452 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:57.829948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:47325)
0809 04:37:57.830285 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:57.830348 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":83,"threads_started":1}
I20260809 04:37:58.019318 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:58.018843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:59021)
0809 04:37:58.019147 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:58.019201 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":78,"threads_started":1}
I20260809 04:37:58.215893 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:58.215478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:34739)
0809 04:37:58.215736 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:58.215797 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260809 04:37:58.831668 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:58.831207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:44333)
0809 04:37:58.831508 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:58.831568 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260809 04:37:59.020460 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:59.020060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:42549)
0809 04:37:59.020307 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:59.020363 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260809 04:37:59.217087 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:59.216657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:56533)
0809 04:37:59.216930 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:59.216989 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260809 04:37:59.833021 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:37:59.832592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:57527)
0809 04:37:59.832856 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 04:37:59.832916 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260809 04:38:00.021708 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:00.021287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:45727)
0809 04:38:00.021549 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:00.021608 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260809 04:38:00.218261 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:00.217830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:57841)
0809 04:38:00.218105 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:00.218162 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260809 04:38:00.597373 10598 raft_consensus.cc:482] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:38:00.597502 10598 raft_consensus.cc:504] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:38:00.597872 10598 leader_election.cc:288] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2735617de1401987f7556909143af3 (127.19.92.3:42307), e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
W20260809 04:38:00.598582 9969 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2735617de1401987f7556909143af3 (127.19.92.3:42307): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
W20260809 04:38:00.598703 9971 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
I20260809 04:38:00.598793 9971 leader_election.cc:302] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519; no voters: ce2735617de1401987f7556909143af3, e7613783bd524ea1851327bc707711e8
I20260809 04:38:00.598930 10598 raft_consensus.cc:2726] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:38:00.834277 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:00.833828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:59127)
0809 04:38:00.834114 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:00.834176 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260809 04:38:01.022912 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:01.022462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:47117)
0809 04:38:01.022744 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:01.022811 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260809 04:38:01.219563 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:01.219097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:44617)
0809 04:38:01.219364 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:01.219458 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260809 04:38:01.835587 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:01.835127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:39899)
0809 04:38:01.835397 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:01.835459 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260809 04:38:02.024207 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:02.023724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:38047)
0809 04:38:02.024049 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:02.024109 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":75,"threads_started":1}
I20260809 04:38:02.220914 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:02.220335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:50935)
0809 04:38:02.220727 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:02.220802 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":97,"threads_started":1}
I20260809 04:38:02.836721 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:02.836337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:59315)
0809 04:38:02.836570 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:02.836624 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260809 04:38:03.025432 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:03.024994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:37437)
0809 04:38:03.025269 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:03.025324 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260809 04:38:03.204761 10614 raft_consensus.cc:482] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:38:03.204901 10614 raft_consensus.cc:504] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:38:03.205374 10614 leader_election.cc:288] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2735617de1401987f7556909143af3 (127.19.92.3:42307), e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
W20260809 04:38:03.206135 9969 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2735617de1401987f7556909143af3 (127.19.92.3:42307): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
W20260809 04:38:03.206225 9971 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
I20260809 04:38:03.206333 9971 leader_election.cc:302] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519; no voters: ce2735617de1401987f7556909143af3, e7613783bd524ea1851327bc707711e8
I20260809 04:38:03.206506 10614 raft_consensus.cc:2726] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:38:03.222083 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:03.221625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:46429)
0809 04:38:03.221925 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:03.221985 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260809 04:38:03.838104 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:03.837620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:37655)
0809 04:38:03.837924 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:03.838004 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":88,"threads_started":1}
I20260809 04:38:04.026746 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:04.026263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:35711)
0809 04:38:04.026571 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:04.026639 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
I20260809 04:38:04.223352 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:04.222896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:52869)
0809 04:38:04.223185 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:04.223251 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260809 04:38:04.839571 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:04.839131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:58475)
0809 04:38:04.839402 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:04.839465 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260809 04:38:05.028095 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:05.027600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:55271)
0809 04:38:05.027912 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:05.027996 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
I20260809 04:38:05.224712 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:05.224237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:33367)
0809 04:38:05.224544 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:05.224611 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":101,"threads_started":1}
I20260809 04:38:05.840772 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:05.840383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:40131)
0809 04:38:05.840617 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:05.840671 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260809 04:38:05.854318 10630 raft_consensus.cc:482] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:38:05.854471 10630 raft_consensus.cc:504] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2735617de1401987f7556909143af3" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 42307 } } peers { permanent_uuid: "e7613783bd524ea1851327bc707711e8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33223 } } peers { permanent_uuid: "17cd6219ec254798858cbc9739ae5519" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 44411 } }
I20260809 04:38:05.854897 10630 leader_election.cc:288] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2735617de1401987f7556909143af3 (127.19.92.3:42307), e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223)
W20260809 04:38:05.855700 9969 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2735617de1401987f7556909143af3 (127.19.92.3:42307): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
W20260809 04:38:05.855808 9971 leader_election.cc:341] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7613783bd524ea1851327bc707711e8 (127.19.92.2:33223): Not found: Tablet not found: 09fa5219cbcc43fc995645583060658b
I20260809 04:38:05.855904 9971 leader_election.cc:302] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17cd6219ec254798858cbc9739ae5519; no voters: ce2735617de1401987f7556909143af3, e7613783bd524ea1851327bc707711e8
I20260809 04:38:05.856071 10630 raft_consensus.cc:2726] T 09fa5219cbcc43fc995645583060658b P 17cd6219ec254798858cbc9739ae5519 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:38:06.029393 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:06.028875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:57649)
0809 04:38:06.029231 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:06.029292 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":80,"threads_started":1}
I20260809 04:38:06.225976 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:06.225568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:45751)
0809 04:38:06.225819 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:06.225876 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260809 04:38:06.841957 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:06.841545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:60821)
0809 04:38:06.841796 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:06.841854 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
I20260809 04:38:07.030616 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:07.030177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:44493)
0809 04:38:07.030432 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:07.030517 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260809 04:38:07.227183 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:07.226771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:33841)
0809 04:38:07.227031 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:07.227085 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260809 04:38:07.843171 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:07.842715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:36715)
0809 04:38:07.843006 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:07.843070 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260809 04:38:08.031922 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:08.031444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:56815)
0809 04:38:08.031745 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:08.031810 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20260809 04:38:08.228428 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:08.227979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:42917)
0809 04:38:08.228267 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:08.228329 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260809 04:38:08.844448 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:08.843982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:46059)
0809 04:38:08.844265 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:08.844330 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260809 04:38:09.033365 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:09.032830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:34291)
0809 04:38:09.033124 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:09.033230 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
I20260809 04:38:09.229671 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:09.229225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:44855)
0809 04:38:09.229511 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:09.229571 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260809 04:38:09.845814 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:09.845408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.1:40849)
0809 04:38:09.845652 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:09.845714 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260809 04:38:10.034778 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:10.034337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.2:49431)
0809 04:38:10.034623 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:10.034679 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260809 04:38:10.220948 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 9896
I20260809 04:38:10.232086 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:10.231640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41143 (local address 127.19.92.3:49143)
0809 04:38:10.231922 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:10.231988 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260809 04:38:10.236852 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 10160
I20260809 04:38:10.247788 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 10348
2026-08-09T04:38:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19920 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:38:10 dist-test-slave-rmhg krb5kdc[10658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:38:10 dist-test-slave-rmhg krb5kdc[10658](info): set up 2 sockets
Aug 09 04:38:10 dist-test-slave-rmhg krb5kdc[10658](info): commencing operation
krb5kdc: starting...
W20260809 04:38:11.385571 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:38:11 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:38:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:38:11.513003 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41319
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:38:11.629204 10675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:11.629503 10675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:11.629565 10675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:11.633603 10675 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 04:38:11.633657 10675 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 04:38:11.633685 10675 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 04:38:11.633702 10675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:38:11.633728 10675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:11.633761 10675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:11.633788 10675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:38:11.633813 10675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:38:11.639168 10675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:11.640301 10675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:11.641409 10675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:38:11.650305 10685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:11.650266 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:11.650285 10675 server_base.cc:1034] running on GCE node
W20260809 04:38:11.650257 10684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:11.651157 10675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:11.652302 10675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:11.653470 10675 hybrid_clock.cc:648] HybridClock initialized: now 1786250291653449 us; error 34 us; skew 500 ppm
Aug 09 04:38:11 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250291, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:11.656162 10675 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:38:11.657127 10675 webserver.cc:460] Webserver started at http://127.19.92.62:41705/ using document root <none> and password file <none>
I20260809 04:38:11.657752 10675 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:11.657819 10675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:11.658053 10675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:38:11.660243 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "92e181e571a24d809eb7abf7e6c56875"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "89045383d6804a5ca6a813b90597cc15"
server_key_iv: "013f35c31926559fbbc7dc82ab51356e"
server_key_version: "encryptionkey@0"
I20260809 04:38:11.660830 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "92e181e571a24d809eb7abf7e6c56875"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "89045383d6804a5ca6a813b90597cc15"
server_key_iv: "013f35c31926559fbbc7dc82ab51356e"
server_key_version: "encryptionkey@0"
I20260809 04:38:11.664829 10675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 04:38:11.667369 10697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:11.668663 10675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:38:11.668776 10675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "92e181e571a24d809eb7abf7e6c56875"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "89045383d6804a5ca6a813b90597cc15"
server_key_iv: "013f35c31926559fbbc7dc82ab51356e"
server_key_version: "encryptionkey@0"
I20260809 04:38:11.668881 10675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:11.689968 10675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:11.693053 10675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:11.693266 10675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:11.700675 10675 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41319
I20260809 04:38:11.700954 10775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41319 every 8 connection(s)
I20260809 04:38:11.701762 10675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:38:11.704782 10776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:38:11.710698 10776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Bootstrap starting.
I20260809 04:38:11.710778 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 10675
I20260809 04:38:11.710919 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:38:11.711237 19824 external_mini_cluster.cc:1428] Setting key a32e79a9fcaa60768c8239932fbde63f
I20260809 04:38:11.713548 10776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Neither blocks nor log segments found. Creating new log.
I20260809 04:38:11.714313 10776 log.cc:824] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:11.716439 10776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: No bootstrap required, opened a new log
Aug 09 04:38:11 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:38:11.720964 10776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:11.721288 10776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:38:11.721371 10776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e181e571a24d809eb7abf7e6c56875, State: Initialized, Role: FOLLOWER
I20260809 04:38:11.721885 10776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:11.722057 10776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:38:11.722133 10776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:38:11.722215 10776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:38:11.723364 10776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:11.723731 10776 leader_election.cc:302] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e181e571a24d809eb7abf7e6c56875; no voters:
I20260809 04:38:11.724119 10776 leader_election.cc:288] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:38:11.724213 10786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:38:11.724457 10786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 1 LEADER]: Becoming Leader. State: Replica: 92e181e571a24d809eb7abf7e6c56875, State: Running, Role: LEADER
I20260809 04:38:11.724843 10786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:11.725489 10776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:38:11.726697 10788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92e181e571a24d809eb7abf7e6c56875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } } }
I20260809 04:38:11.726745 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:11.712651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.19.92.62:41319)
0809 04:38:11.713105 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:11.713122 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:11.713185 (+ 63us) server_negotiation.cc:407] Connection header received
0809 04:38:11.714027 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:11.714050 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:11.714404 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:11.714762 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:11.715499 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:11.716472 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:11.717665 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:11.718170 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:11.718300 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:11.721177 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:11.721185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:11.721203 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:11.721236 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:11.723217 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:11.723659 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:11.723665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:11.723671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:11.723818 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:11.724158 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:11.724162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:11.724163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:11.724782 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:11.724989 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:11.725341 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 04:38:11.725667 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":142,"threads_started":1}
I20260809 04:38:11.726830 10788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: This master's current role is: LEADER
I20260809 04:38:11.727886 10790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e181e571a24d809eb7abf7e6c56875. Latest consensus state: current_term: 1 leader_uuid: "92e181e571a24d809eb7abf7e6c56875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } } }
I20260809 04:38:11.728055 10790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: This master's current role is: LEADER
W20260809 04:38:11.730331 10808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:38:11.730396 10808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:38:11.730475 10800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:38:11.731175 10800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:38:11.736686 10800 catalog_manager.cc:1347] Generated new cluster ID: 1b9363a75ab5494882b8bbcc4e4ea6c8
I20260809 04:38:11.736763 10800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:38:11.753364 10800 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:38:11.754731 10800 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:38:11.762547 10800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Generated new TSK 0
I20260809 04:38:11.763278 10800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:38:11.805048 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41319
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:38:11.941845 10816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:11.942150 10816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:11.942194 10816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:11.946499 10816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:11.946584 10816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:11.946650 10816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:38:11.952338 10816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:11.953464 10816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:11.954464 10816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:38:11.964021 10816 server_base.cc:1034] running on GCE node
W20260809 04:38:11.963982 10826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:11.963922 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:11.963922 10828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:11.964547 10816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:11.965134 10816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:11.966308 10816 hybrid_clock.cc:648] HybridClock initialized: now 1786250291966289 us; error 31 us; skew 500 ppm
Aug 09 04:38:11 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250291, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:11.968964 10816 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:38:11.969888 10816 webserver.cc:460] Webserver started at http://127.19.92.1:35551/ using document root <none> and password file <none>
I20260809 04:38:11.970425 10816 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:11.970489 10816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:11.970682 10816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:38:11.972832 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "9409bf8060a343788fa2e514a2935cc0"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "5f017bc88f01d950154b00f3c36aab22"
server_key_iv: "d6633e58c7fcc2f1c870410773812d12"
server_key_version: "encryptionkey@0"
I20260809 04:38:11.973436 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "9409bf8060a343788fa2e514a2935cc0"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "5f017bc88f01d950154b00f3c36aab22"
server_key_iv: "d6633e58c7fcc2f1c870410773812d12"
server_key_version: "encryptionkey@0"
I20260809 04:38:11.977336 10816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 04:38:11.980080 10835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:11.981451 10816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:38:11.981572 10816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "9409bf8060a343788fa2e514a2935cc0"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "5f017bc88f01d950154b00f3c36aab22"
server_key_iv: "d6633e58c7fcc2f1c870410773812d12"
server_key_version: "encryptionkey@0"
I20260809 04:38:11.981683 10816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:12.016418 10816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:12.019989 10816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:12.020190 10816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:12.020836 10816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:12.021751 10816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:38:12.021803 10816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:12.021859 10816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:38:12.021888 10816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:12.030514 10816 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:45977
I20260809 04:38:12.030548 11002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:45977 every 8 connection(s)
I20260809 04:38:12.031405 10816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:38:12.033860 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 10816
I20260809 04:38:12.034014 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:38:12.034346 19824 external_mini_cluster.cc:1428] Setting key 752b51e2a52bf37a3f612ad9e9408108
Aug 09 04:38:12 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250291, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:38:12.045572 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.033243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:53563 (local address 127.19.92.62:41319)
0809 04:38:12.033396 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.033401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.034587 (+ 1186us) server_negotiation.cc:407] Connection header received
0809 04:38:12.035580 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.035584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.035635 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.035710 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:12.036886 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.037470 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.039060 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.039291 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.039371 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.042057 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:12.042060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:12.042062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:12.042087 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:12.043500 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:12.043987 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:12.043990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:12.043991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:12.044073 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:12.044386 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:12.044389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:12.044391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:12.044633 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:12.044728 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.045252 (+ 524us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.045388 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:38:12.046380 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.033616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:53563)
0809 04:38:12.034382 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.034479 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:12.035373 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:12.035898 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:12.035905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:12.036255 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:12.036749 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.036763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.037593 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.037596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.038929 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.038942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.039448 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.039453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.039640 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.040220 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:12.040239 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:12.041891 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:12.043612 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:12.043618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:12.043635 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:12.043885 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:12.044155 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:12.044159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:12.044161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:12.044298 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:12.044738 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:12.044745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:12.045004 (+ 259us) client_negotiation.cc:769] Sending connection context
0809 04:38:12.045205 (+ 201us) client_negotiation.cc:240] Negotiation successful
0809 04:38:12.045538 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:38:12.047684 11004 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41319
I20260809 04:38:12.047924 11004 heartbeater.cc:464] Registering TS with master...
I20260809 04:38:12.048435 11004 heartbeater.cc:511] Master 127.19.92.62:41319 requested a full tablet report, sending...
I20260809 04:38:12.050119 10723 ts_manager.cc:194] Registered new tserver with Master: 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977)
I20260809 04:38:12.051265 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:53563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:38:12.064574 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41319
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:38:12.178649 11011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:12.179000 11011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:12.179070 11011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:12.183372 11011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:12.183445 11011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:12.183531 11011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:38:12.189177 11011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:12.190325 11011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:12.191380 11011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:38:12.201265 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:12.201344 11020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:12.201336 11011 server_base.cc:1034] running on GCE node
W20260809 04:38:12.201273 11016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:12.201965 11011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:12.202584 11011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:12.203743 11011 hybrid_clock.cc:648] HybridClock initialized: now 1786250292203723 us; error 33 us; skew 500 ppm
Aug 09 04:38:12 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250292, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:12.206496 11011 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:38:12.207405 11011 webserver.cc:460] Webserver started at http://127.19.92.2:42807/ using document root <none> and password file <none>
I20260809 04:38:12.207950 11011 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:12.208029 11011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:12.208235 11011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:38:12.210333 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "2acc874aad744fe399f748363809cde8"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f723354b1cebd840acb1da3112b6b712"
server_key_iv: "0e7ad4b30f2383ee7c966ef493763785"
server_key_version: "encryptionkey@0"
I20260809 04:38:12.210867 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "2acc874aad744fe399f748363809cde8"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f723354b1cebd840acb1da3112b6b712"
server_key_iv: "0e7ad4b30f2383ee7c966ef493763785"
server_key_version: "encryptionkey@0"
I20260809 04:38:12.214540 11011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:38:12.217113 11029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:12.218495 11011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:38:12.218606 11011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "2acc874aad744fe399f748363809cde8"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f723354b1cebd840acb1da3112b6b712"
server_key_iv: "0e7ad4b30f2383ee7c966ef493763785"
server_key_version: "encryptionkey@0"
I20260809 04:38:12.218731 11011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:12.230044 11011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:12.232791 11011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:12.232960 11011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:12.233590 11011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:12.234447 11011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:38:12.234508 11011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:12.234561 11011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:38:12.234586 11011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:12.242908 11011 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:36937
I20260809 04:38:12.242933 11195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:36937 every 8 connection(s)
I20260809 04:38:12.243804 11011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:38:12.252243 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 11011
I20260809 04:38:12.252384 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:38:12.252730 19824 external_mini_cluster.cc:1428] Setting key dd091f6136c1f26a869bf01b389c9d38
Aug 09 04:38:12 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250292, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:38:12.258625 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.245654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:52817 (local address 127.19.92.62:41319)
0809 04:38:12.245783 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.245787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.246875 (+ 1088us) server_negotiation.cc:407] Connection header received
0809 04:38:12.247803 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.247806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.247857 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.247950 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:12.249267 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.249832 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.250959 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.251175 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.251243 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.253851 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:12.253854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:12.253856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:12.253882 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:12.256162 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:12.256861 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:12.256864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:12.256865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:12.256948 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:12.257342 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:12.257345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:12.257346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:12.257565 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:12.257645 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.258361 (+ 716us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.258493 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:38:12.259915 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.246043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:52817)
0809 04:38:12.246707 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.246749 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:12.247567 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:12.248192 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:12.248200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:12.248564 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:12.249087 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.249103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.249939 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.249941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.250862 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.250872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.251283 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.251285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.251425 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.252017 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:12.252038 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:12.253686 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:12.256304 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:12.256312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:12.256337 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:12.256764 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:12.257028 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:12.257031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:12.257033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:12.257253 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:12.257668 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:12.257675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:12.258064 (+ 389us) client_negotiation.cc:769] Sending connection context
0809 04:38:12.258280 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 04:38:12.258678 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":114,"threads_started":1}
I20260809 04:38:12.261229 11196 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41319
I20260809 04:38:12.261480 11196 heartbeater.cc:464] Registering TS with master...
I20260809 04:38:12.262022 11196 heartbeater.cc:511] Master 127.19.92.62:41319 requested a full tablet report, sending...
I20260809 04:38:12.263135 10723 ts_manager.cc:194] Registered new tserver with Master: 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937)
I20260809 04:38:12.263724 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:52817
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:38:12.282191 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41319
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:38:12.401043 11205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:12.401345 11205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:12.401423 11205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:12.405434 11205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:12.405498 11205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:12.405581 11205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:38:12.410846 11205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:12.411906 11205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:12.412931 11205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:38:12.422724 11211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:12.422724 11214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:12.422789 11212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:12.422982 11205 server_base.cc:1034] running on GCE node
I20260809 04:38:12.423493 11205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:12.424189 11205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:12.425344 11205 hybrid_clock.cc:648] HybridClock initialized: now 1786250292425323 us; error 32 us; skew 500 ppm
Aug 09 04:38:12 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250292, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:12.430934 11205 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:38:12.431912 11205 webserver.cc:460] Webserver started at http://127.19.92.3:33855/ using document root <none> and password file <none>
I20260809 04:38:12.432483 11205 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:12.432576 11205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:12.432796 11205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:38:12.435050 11205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "7560ad4155a64a8cb492e0b2c476e7e4"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f2ee41c5a86dad74c9c8bd3b66f493ac"
server_key_iv: "362e48687d753e0f0e7e9a79b1f3d6ad"
server_key_version: "encryptionkey@0"
I20260809 04:38:12.435608 11205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "7560ad4155a64a8cb492e0b2c476e7e4"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f2ee41c5a86dad74c9c8bd3b66f493ac"
server_key_iv: "362e48687d753e0f0e7e9a79b1f3d6ad"
server_key_version: "encryptionkey@0"
I20260809 04:38:12.439487 11205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:38:12.442034 11227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:12.443279 11205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:38:12.443399 11205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "7560ad4155a64a8cb492e0b2c476e7e4"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f2ee41c5a86dad74c9c8bd3b66f493ac"
server_key_iv: "362e48687d753e0f0e7e9a79b1f3d6ad"
server_key_version: "encryptionkey@0"
I20260809 04:38:12.443516 11205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:12.486042 11205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:12.489503 11205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:12.489737 11205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:12.490396 11205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:12.491291 11205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:38:12.491343 11205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:12.491400 11205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:38:12.491427 11205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:12.500722 11205 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:43081
I20260809 04:38:12.500749 11398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:43081 every 8 connection(s)
I20260809 04:38:12.501715 11205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:38:12.510272 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 11205
I20260809 04:38:12.510391 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:38:12.510689 19824 external_mini_cluster.cc:1428] Setting key d8c46bef8247875ee3e297114cdeb986
Aug 09 04:38:12 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250292, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:38:12.515353 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.503593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:42697 (local address 127.19.92.62:41319)
0809 04:38:12.503745 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.503749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.504660 (+ 911us) server_negotiation.cc:407] Connection header received
0809 04:38:12.506022 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.506027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.506088 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.506166 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:12.507443 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.507984 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.509186 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.509415 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.509506 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.512149 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:12.512151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:12.512154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:12.512181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:12.513574 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:12.514063 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:12.514065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:12.514066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:12.514150 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:12.514407 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:12.514409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:12.514410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:12.514584 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:12.514659 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.515124 (+ 465us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.515231 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:38:12.516227 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.504010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:42697)
0809 04:38:12.504485 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.504551 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:12.505782 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:12.506366 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:12.506373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:12.506782 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:12.507291 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.507306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.508115 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.508118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.509036 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.509047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.509569 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.509573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.509741 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.510330 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:12.510354 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:12.511959 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:12.513691 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:12.513697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:12.513711 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:12.513970 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:12.514213 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:12.514215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:12.514216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:12.514346 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:12.514658 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:12.514663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:12.514913 (+ 250us) client_negotiation.cc:769] Sending connection context
0809 04:38:12.515083 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 04:38:12.515352 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260809 04:38:12.517390 11400 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41319
I20260809 04:38:12.517644 11400 heartbeater.cc:464] Registering TS with master...
I20260809 04:38:12.518136 11400 heartbeater.cc:511] Master 127.19.92.62:41319 requested a full tablet report, sending...
I20260809 04:38:12.519151 10723 ts_manager.cc:194] Registered new tserver with Master: 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081)
I20260809 04:38:12.519711 10723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:42697
I20260809 04:38:12.524467 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:38:12.533938 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.526404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.19.92.62:41319)
0809 04:38:12.526521 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.526524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.526804 (+ 280us) server_negotiation.cc:407] Connection header received
0809 04:38:12.526934 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.526937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.526985 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.527046 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:12.527838 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.528368 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.529240 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.529469 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.529558 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.530968 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:12.530972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:12.530975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:12.531002 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:12.532178 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:12.532647 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:12.532650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:12.532652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:12.532738 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:12.532985 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:12.532989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:12.532991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:12.533288 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:12.533413 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.533657 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.533777 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 04:38:12.537040 10723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:38:12.540354 10723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:38:12.555393 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.549560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43081 (local address 127.0.0.1:35238)
0809 04:38:12.550614 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.550627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:12.550710 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:12.551263 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:12.551270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:12.551290 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:12.551444 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.551450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.552904 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.552907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.554040 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.554048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.554841 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.554844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.554950 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.555000 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 04:38:12.555113 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 04:38:12.555235 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260809 04:38:12.555428 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.550203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.19.92.3:43081)
0809 04:38:12.550762 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.550774 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.550795 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:38:12.550859 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.550864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.551031 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.551153 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:12.551590 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.552760 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.554158 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.554728 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.554829 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.554987 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.555167 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.555275 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":164,"threads_started":1}
I20260809 04:38:12.555539 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.549570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.19.92.2:36937)
0809 04:38:12.550316 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.550329 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.550565 (+ 236us) server_negotiation.cc:407] Connection header received
0809 04:38:12.550755 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.550761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.550981 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.551119 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:12.551553 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.552384 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.553937 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.554765 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.554872 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.555014 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.555298 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.555391 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":117,"threads_started":1}
I20260809 04:38:12.555891 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.550217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.19.92.1:45977)
0809 04:38:12.550934 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.550939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.551137 (+ 198us) server_negotiation.cc:407] Connection header received
0809 04:38:12.551240 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.551245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.551430 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.551561 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:12.551747 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.552809 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.554224 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.554959 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.555061 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.555183 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.555704 (+ 521us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.555785 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":81,"threads_started":1}
I20260809 04:38:12.555609 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.549772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45977 (local address 127.0.0.1:51224)
0809 04:38:12.551040 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.551050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:12.551138 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:12.551557 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:12.551559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:12.551575 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:12.551647 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.551652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.553097 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.553100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.554089 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.554100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.555245 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.555249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.555331 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.555354 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:38:12.555421 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 04:38:12.555487 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260809 04:38:12.555864 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.549399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:36937 (local address 127.0.0.1:60592)
0809 04:38:12.550080 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.550427 (+ 347us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:12.550570 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:12.551330 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:12.551333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:12.551349 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:12.551431 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.551435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.552513 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.552517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.553809 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.553820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.555063 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.555067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.555150 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.555170 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 04:38:12.555672 (+ 502us) client_negotiation.cc:240] Negotiation successful
0809 04:38:12.555750 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":81,"threads_started":1}
I20260809 04:38:12.558854 10898 tablet_service.cc:1467] Processing CreateTablet for tablet a413eb2e976d4d06a8f8234a1b6f32cb (DEFAULT_TABLE table=test-table [id=8a19b451a714489ab146a0f360a133c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:38:12.558949 11302 tablet_service.cc:1467] Processing CreateTablet for tablet a413eb2e976d4d06a8f8234a1b6f32cb (DEFAULT_TABLE table=test-table [id=8a19b451a714489ab146a0f360a133c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:38:12.558949 11096 tablet_service.cc:1467] Processing CreateTablet for tablet a413eb2e976d4d06a8f8234a1b6f32cb (DEFAULT_TABLE table=test-table [id=8a19b451a714489ab146a0f360a133c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:38:12.559870 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413eb2e976d4d06a8f8234a1b6f32cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:38:12.559983 11096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413eb2e976d4d06a8f8234a1b6f32cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:38:12.559995 11302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413eb2e976d4d06a8f8234a1b6f32cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:38:12.566361 11423 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Bootstrap starting.
I20260809 04:38:12.567552 11424 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: Bootstrap starting.
I20260809 04:38:12.567991 11426 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: Bootstrap starting.
I20260809 04:38:12.568943 11423 tablet_bootstrap.cc:654] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Neither blocks nor log segments found. Creating new log.
I20260809 04:38:12.569922 11423 log.cc:824] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:12.570662 11426 tablet_bootstrap.cc:654] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: Neither blocks nor log segments found. Creating new log.
I20260809 04:38:12.571511 11424 tablet_bootstrap.cc:654] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: Neither blocks nor log segments found. Creating new log.
I20260809 04:38:12.571532 11426 log.cc:824] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:12.572324 11423 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: No bootstrap required, opened a new log
I20260809 04:38:12.572551 11423 ts_tablet_manager.cc:1397] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 04:38:12.572521 11424 log.cc:824] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:12.574028 11426 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: No bootstrap required, opened a new log
I20260809 04:38:12.574249 11426 ts_tablet_manager.cc:1397] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:38:12.574435 11424 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: No bootstrap required, opened a new log
I20260809 04:38:12.574615 11424 ts_tablet_manager.cc:1397] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 04:38:12.577576 11424 raft_consensus.cc:348] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.577579 11423 raft_consensus.cc:348] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.577881 11424 raft_consensus.cc:374] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:38:12.577890 11423 raft_consensus.cc:374] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:38:12.577946 11423 raft_consensus.cc:729] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9409bf8060a343788fa2e514a2935cc0, State: Initialized, Role: FOLLOWER
I20260809 04:38:12.577963 11424 raft_consensus.cc:729] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2acc874aad744fe399f748363809cde8, State: Initialized, Role: FOLLOWER
I20260809 04:38:12.578393 11426 raft_consensus.cc:348] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.578598 11423 consensus_queue.cc:260] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.578751 11426 raft_consensus.cc:374] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:38:12.578835 11426 raft_consensus.cc:729] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7560ad4155a64a8cb492e0b2c476e7e4, State: Initialized, Role: FOLLOWER
I20260809 04:38:12.578764 11424 consensus_queue.cc:260] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.579348 11426 consensus_queue.cc:260] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.579886 11423 ts_tablet_manager.cc:1428] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 04:38:12.579916 11424 ts_tablet_manager.cc:1428] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:38:12.580214 11400 heartbeater.cc:503] Master 127.19.92.62:41319 was elected leader, sending a full tablet report...
I20260809 04:38:12.580293 11004 heartbeater.cc:503] Master 127.19.92.62:41319 was elected leader, sending a full tablet report...
I20260809 04:38:12.580403 11426 ts_tablet_manager.cc:1428] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:38:12.581293 11196 heartbeater.cc:503] Master 127.19.92.62:41319 was elected leader, sending a full tablet report...
I20260809 04:38:12.599526 11434 raft_consensus.cc:482] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:38:12.599733 11434 raft_consensus.cc:504] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.600878 11434 leader_election.cc:288] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081), 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977)
I20260809 04:38:12.604307 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.601224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:50675 (local address 127.19.92.3:43081)
0809 04:38:12.601359 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.601364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.601382 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:38:12.601440 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.601442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.601498 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.601576 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:12.601888 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.602390 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.603400 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.603797 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.603888 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.603956 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.604134 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.604204 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:38:12.604310 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.601106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43081 (local address 127.19.92.2:50675)
0809 04:38:12.601234 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.601248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:12.601338 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:12.601609 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:12.601612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:12.601632 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:12.601769 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.601775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.602491 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.602495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.603271 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.603280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.603904 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.603907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.604019 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.604038 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:38:12.604103 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 04:38:12.604173 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1,"spinlock_wait_cycles":6016}
I20260809 04:38:12.604583 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.601285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:52105 (local address 127.19.92.1:45977)
0809 04:38:12.601394 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.601398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.601667 (+ 269us) server_negotiation.cc:407] Connection header received
0809 04:38:12.601796 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.601799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.601843 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.601901 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:12.602153 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.602652 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.603652 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.604065 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.604155 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.604214 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.604379 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.604471 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:38:12.604580 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.601106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45977 (local address 127.19.92.2:52105)
0809 04:38:12.601577 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.601587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:12.601702 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:12.601922 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:12.601925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:12.601939 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:12.602057 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.602062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.602758 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.602761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.603538 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:12.603546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.604178 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.604182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:12.604275 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.604294 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:38:12.604356 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 04:38:12.604435 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":128,"spinlock_wait_cycles":29824,"thread_start_us":133,"threads_started":1}
I20260809 04:38:12.605086 11332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413eb2e976d4d06a8f8234a1b6f32cb" candidate_uuid: "2acc874aad744fe399f748363809cde8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" is_pre_election: true
I20260809 04:38:12.605315 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413eb2e976d4d06a8f8234a1b6f32cb" candidate_uuid: "2acc874aad744fe399f748363809cde8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9409bf8060a343788fa2e514a2935cc0" is_pre_election: true
I20260809 04:38:12.605440 11332 raft_consensus.cc:2454] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2acc874aad744fe399f748363809cde8 in term 0.
I20260809 04:38:12.605592 10929 raft_consensus.cc:2454] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2acc874aad744fe399f748363809cde8 in term 0.
I20260809 04:38:12.605914 11036 leader_election.cc:302] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4; no voters:
I20260809 04:38:12.606163 11434 raft_consensus.cc:2781] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:38:12.606241 11434 raft_consensus.cc:482] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:38:12.606292 11434 raft_consensus.cc:3037] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:38:12.607332 11434 raft_consensus.cc:504] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.607663 11434 leader_election.cc:288] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [CANDIDATE]: Term 1 election: Requested vote from peers 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081), 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977)
I20260809 04:38:12.607991 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413eb2e976d4d06a8f8234a1b6f32cb" candidate_uuid: "2acc874aad744fe399f748363809cde8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9409bf8060a343788fa2e514a2935cc0"
I20260809 04:38:12.608026 11332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413eb2e976d4d06a8f8234a1b6f32cb" candidate_uuid: "2acc874aad744fe399f748363809cde8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7560ad4155a64a8cb492e0b2c476e7e4"
I20260809 04:38:12.608125 10929 raft_consensus.cc:3037] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:38:12.608143 11332 raft_consensus.cc:3037] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:38:12.609251 10929 raft_consensus.cc:2454] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2acc874aad744fe399f748363809cde8 in term 1.
I20260809 04:38:12.609292 11332 raft_consensus.cc:2454] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2acc874aad744fe399f748363809cde8 in term 1.
I20260809 04:38:12.609640 11036 leader_election.cc:302] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4, 9409bf8060a343788fa2e514a2935cc0; no voters:
I20260809 04:38:12.609874 11434 raft_consensus.cc:2781] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:38:12.610077 11434 raft_consensus.cc:686] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 1 LEADER]: Becoming Leader. State: Replica: 2acc874aad744fe399f748363809cde8, State: Running, Role: LEADER
I20260809 04:38:12.610426 11434 consensus_queue.cc:237] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:12.614001 10723 catalog_manager.cc:5495] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2acc874aad744fe399f748363809cde8 (127.19.92.2). New cstate: current_term: 1 leader_uuid: "2acc874aad744fe399f748363809cde8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } health_report { overall_health: UNKNOWN } } }
I20260809 04:38:12.635245 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.631652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.19.92.2:36937)
0809 04:38:12.631767 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:12.631770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:12.631783 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:38:12.631893 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:12.631896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:12.631970 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:12.632069 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:38:12.632345 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.632858 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.633844 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:12.634075 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:12.634168 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:12.634435 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:38:12.634941 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:38:12.635045 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:12.635076 (+ 31us) server_negotiation.cc:299] Negotiation successful
0809 04:38:12.635132 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:38:12.641278 10929 raft_consensus.cc:1261] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Refusing update from remote peer 2acc874aad744fe399f748363809cde8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:38:12.641278 11332 raft_consensus.cc:1261] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 1 FOLLOWER]: Refusing update from remote peer 2acc874aad744fe399f748363809cde8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:38:12.642000 11434 consensus_queue.cc:1035] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:38:12.642194 11438 consensus_queue.cc:1035] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:38:12.653043 11443 mvcc.cc:205] Tried to move back new op lower bound from 7316481198650880000 to 7316481198535630848. Current Snapshot: MvccSnapshot[applied={T|T < 7316481198650880000}]
I20260809 04:38:12.658035 10721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:38:12.658205 10721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:38:12.660949 10721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Sending DeleteTablet for 3 replicas of tablet a413eb2e976d4d06a8f8234a1b6f32cb
I20260809 04:38:12.661849 11302 tablet_service.cc:1514] Processing DeleteTablet for tablet a413eb2e976d4d06a8f8234a1b6f32cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:38:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:35238
I20260809 04:38:12.661849 11096 tablet_service.cc:1514] Processing DeleteTablet for tablet a413eb2e976d4d06a8f8234a1b6f32cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:38:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:60592
I20260809 04:38:12.662474 11454 tablet_replica.cc:330] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: stopping tablet replica
I20260809 04:38:12.662782 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 10816
I20260809 04:38:12.662720 10898 tablet_service.cc:1514] Processing DeleteTablet for tablet a413eb2e976d4d06a8f8234a1b6f32cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:38:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:51224
I20260809 04:38:12.662902 11454 raft_consensus.cc:2229] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:38:12.663228 11454 raft_consensus.cc:2258] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:38:12.669368 11454 ts_tablet_manager.cc:1905] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:38:12.671787 11455 tablet_replica.cc:330] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: stopping tablet replica
I20260809 04:38:12.672128 11455 raft_consensus.cc:2229] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:38:12.672534 11455 raft_consensus.cc:2258] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:38:12.676801 11455 ts_tablet_manager.cc:1905] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:38:12.679466 11455 ts_tablet_manager.cc:1918] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:38:12.679477 11454 ts_tablet_manager.cc:1918] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:38:12.679591 11454 log.cc:1196] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/a413eb2e976d4d06a8f8234a1b6f32cb
I20260809 04:38:12.679682 11455 log.cc:1196] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/a413eb2e976d4d06a8f8234a1b6f32cb
I20260809 04:38:12.680045 11454 ts_tablet_manager.cc:1939] T a413eb2e976d4d06a8f8234a1b6f32cb P 7560ad4155a64a8cb492e0b2c476e7e4: Deleting consensus metadata
I20260809 04:38:12.680104 11455 ts_tablet_manager.cc:1939] T a413eb2e976d4d06a8f8234a1b6f32cb P 2acc874aad744fe399f748363809cde8: Deleting consensus metadata
I20260809 04:38:12.681550 10706 catalog_manager.cc:4841] TS 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081): tablet a413eb2e976d4d06a8f8234a1b6f32cb (table test-table [id=8a19b451a714489ab146a0f360a133c0]) successfully deleted
I20260809 04:38:12.681624 10702 catalog_manager.cc:4841] TS 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937): tablet a413eb2e976d4d06a8f8234a1b6f32cb (table test-table [id=8a19b451a714489ab146a0f360a133c0]) successfully deleted
W20260809 04:38:12.686205 10704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:45977 (error 108)
I20260809 04:38:12.687132 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 11011
I20260809 04:38:12.687362 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:12.687048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45977 (local address 127.0.0.1:51232)
0809 04:38:12.687199 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:12.687272 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 04:38:12.689847 10704 catalog_manager.cc:4584] TS 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a413eb2e976d4d06a8f8234a1b6f32cb: Network error: Client connection negotiation failed: client connection to 127.19.92.1:45977: connect: Connection refused (error 111)
I20260809 04:38:12.697443 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 11205
I20260809 04:38:12.717927 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 10675
I20260809 04:38:32.730633 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41319
--webserver_interface=127.19.92.62
--webserver_port=41705
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:38:32.847623 11462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:32.847888 11462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:32.847934 11462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:32.851922 11462 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 04:38:32.851977 11462 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 04:38:32.851992 11462 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 04:38:32.852005 11462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:38:32.852021 11462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:32.852051 11462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:32.852079 11462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:38:32.852102 11462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:38:32.857350 11462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:32.858377 11462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:32.859400 11462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:38:32.873771 11462 server_base.cc:1034] running on GCE node
W20260809 04:38:32.873723 11471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:32.873720 11469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:32.873720 11475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:32.874718 11462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:32.875958 11462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:32.877123 11462 hybrid_clock.cc:648] HybridClock initialized: now 1786250312877106 us; error 33 us; skew 500 ppm
Aug 09 04:38:32 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250312, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:32.884145 11462 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:38:32.885142 11462 webserver.cc:460] Webserver started at http://127.19.92.62:41705/ using document root <none> and password file <none>
I20260809 04:38:32.885761 11462 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:32.885848 11462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:32.889878 11462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 04:38:32.892009 11486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:32.893270 11462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:38:32.893388 11462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "92e181e571a24d809eb7abf7e6c56875"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "89045383d6804a5ca6a813b90597cc15"
server_key_iv: "013f35c31926559fbbc7dc82ab51356e"
server_key_version: "encryptionkey@0"
I20260809 04:38:32.893989 11462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:32.907903 11462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:32.911042 11462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:32.911248 11462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:32.918634 11462 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41319
I20260809 04:38:32.918992 11554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41319 every 8 connection(s)
I20260809 04:38:32.919853 11462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:38:32.924512 11556 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:38:32.926335 11556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Bootstrap starting.
I20260809 04:38:32.929093 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 11462
I20260809 04:38:32.929593 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:45977
--local_ip_for_outbound_sockets=127.19.92.1
--tserver_master_addrs=127.19.92.62:41319
--webserver_port=35551
--webserver_interface=127.19.92.1
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:38:32.947566 11556 log.cc:824] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:32.959527 11556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:38:32.959976 11556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Bootstrap complete.
I20260809 04:38:32.965389 11556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:32.965914 11556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e181e571a24d809eb7abf7e6c56875, State: Initialized, Role: FOLLOWER
I20260809 04:38:32.966496 11556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:32.966652 11556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:38:32.966724 11556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:38:32.966825 11556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:38:32.969354 11556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:32.969746 11556 leader_election.cc:302] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e181e571a24d809eb7abf7e6c56875; no voters:
I20260809 04:38:32.970090 11556 leader_election.cc:288] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 04:38:32.970213 11562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:38:32.970464 11562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 2 LEADER]: Becoming Leader. State: Replica: 92e181e571a24d809eb7abf7e6c56875, State: Running, Role: LEADER
I20260809 04:38:32.970861 11562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:32.971361 11556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:38:32.972179 11563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "92e181e571a24d809eb7abf7e6c56875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } } }
I20260809 04:38:32.972301 11563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: This master's current role is: LEADER
I20260809 04:38:32.972661 11564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e181e571a24d809eb7abf7e6c56875. Latest consensus state: current_term: 2 leader_uuid: "92e181e571a24d809eb7abf7e6c56875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } } }
I20260809 04:38:32.972808 11564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: This master's current role is: LEADER
I20260809 04:38:32.973112 11576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:38:32.977944 11576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:38:32.978566 11576 catalog_manager.cc:1259] Loaded cluster ID: 1b9363a75ab5494882b8bbcc4e4ea6c8
I20260809 04:38:32.978627 11576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:38:32.981110 11576 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:38:32.981841 11576 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Loaded TSK: 0
I20260809 04:38:32.996080 11576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Generated new TSK 1
I20260809 04:38:32.996378 11576 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:38:33.060703 11559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:33.060961 11559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:33.061021 11559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:33.065054 11559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:33.065141 11559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:33.065253 11559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:38:33.071208 11559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:45977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=35551
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:33.072422 11559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:33.073621 11559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:38:33.083681 11559 server_base.cc:1034] running on GCE node
W20260809 04:38:33.083632 11598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:33.083632 11601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:33.083636 11599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:33.084306 11559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:33.084966 11559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:33.086158 11559 hybrid_clock.cc:648] HybridClock initialized: now 1786250313086143 us; error 56 us; skew 500 ppm
Aug 09 04:38:33 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:33.089105 11559 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:38:33.090026 11559 webserver.cc:460] Webserver started at http://127.19.92.1:35551/ using document root <none> and password file <none>
I20260809 04:38:33.090590 11559 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:33.090677 11559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:33.094770 11559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:38:33.097021 11616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:33.098490 11559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:38:33.098608 11559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "9409bf8060a343788fa2e514a2935cc0"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "5f017bc88f01d950154b00f3c36aab22"
server_key_iv: "d6633e58c7fcc2f1c870410773812d12"
server_key_version: "encryptionkey@0"
I20260809 04:38:33.099148 11559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:33.118614 11559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:33.121770 11559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:33.121955 11559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:33.122570 11559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:33.123653 11627 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 04:38:33.127835 11559 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 04:38:33.127902 11559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 04:38:33.127969 11559 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 04:38:33.129957 11559 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 04:38:33.130007 11559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260809 04:38:33.130282 11627 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Bootstrap starting.
I20260809 04:38:33.139309 11559 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:45977
I20260809 04:38:33.139372 11775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:45977 every 8 connection(s)
I20260809 04:38:33.140259 11559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:38:33.148614 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 11559
I20260809 04:38:33.149091 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:36937
--local_ip_for_outbound_sockets=127.19.92.2
--tserver_master_addrs=127.19.92.62:41319
--webserver_port=42807
--webserver_interface=127.19.92.2
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:38:33.149537 11627 log.cc:824] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:38:33 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:38:33.158896 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.142581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:56205 (local address 127.19.92.62:41319)
0809 04:38:33.143000 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.143021 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.143435 (+ 414us) server_negotiation.cc:407] Connection header received
0809 04:38:33.144865 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.144878 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.145245 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.145540 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:33.147227 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.148404 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.150109 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.150648 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.150821 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.153757 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:33.153763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:33.153777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:33.153809 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:33.155578 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.156112 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.156118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.156122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.156251 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.156588 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.156591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.156592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.156976 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:33.157107 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.157719 (+ 612us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.158010 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":133,"threads_started":1}
I20260809 04:38:33.158910 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.142555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:56205)
0809 04:38:33.143248 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.143308 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:33.144365 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:33.145686 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:33.145697 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:33.146263 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:33.146970 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.147012 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.148606 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.148610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.149964 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.149976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.151034 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.151038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.151222 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.151863 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:33.151884 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:33.153546 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:33.155701 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:33.155708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:33.155723 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:33.155995 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:33.156345 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:33.156348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:33.156351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:33.156494 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:33.157090 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:33.157097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:33.157392 (+ 295us) client_negotiation.cc:769] Sending connection context
0809 04:38:33.157564 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 04:38:33.157940 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":132,"threads_started":1}
I20260809 04:38:33.158927 11627 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:38:33.159466 11627 tablet_bootstrap.cc:492] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Bootstrap complete.
I20260809 04:38:33.160787 11627 ts_tablet_manager.cc:1397] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260809 04:38:33.161561 11776 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41319
I20260809 04:38:33.161798 11776 heartbeater.cc:464] Registering TS with master...
I20260809 04:38:33.162295 11776 heartbeater.cc:511] Master 127.19.92.62:41319 requested a full tablet report, sending...
I20260809 04:38:33.163975 11627 raft_consensus.cc:348] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.164515 11627 raft_consensus.cc:729] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9409bf8060a343788fa2e514a2935cc0, State: Initialized, Role: FOLLOWER
I20260809 04:38:33.164848 11508 ts_manager.cc:194] Registered new tserver with Master: 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977)
I20260809 04:38:33.165051 11627 consensus_queue.cc:260] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.166126 11627 ts_tablet_manager.cc:1428] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 04:38:33.168453 11508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:56205
I20260809 04:38:33.171083 11776 heartbeater.cc:503] Master 127.19.92.62:41319 was elected leader, sending a full tablet report...
Aug 09 04:38:33 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250312, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.1@KRBTEST.COM
I20260809 04:38:33.178234 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.167811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45977 (local address 127.0.0.1:47606)
0809 04:38:33.168562 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.168613 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:33.168758 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:33.169345 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:33.169352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:33.169797 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:33.169934 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.169942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.171262 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.171266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.172039 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.172051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.172629 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.172633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.172746 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.173370 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:33.173388 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:33.174792 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:33.176510 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:33.176519 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:33.176525 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:33.176739 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:33.177038 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:33.177041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:33.177044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:33.177126 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:33.177609 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:33.177615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:33.177739 (+ 124us) client_negotiation.cc:769] Sending connection context
0809 04:38:33.177860 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 04:38:33.178057 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":119,"threads_started":1}
I20260809 04:38:33.178262 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.167892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.19.92.1:45977)
0809 04:38:33.168285 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.168297 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.168712 (+ 415us) server_negotiation.cc:407] Connection header received
0809 04:38:33.168928 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.168934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.169138 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.169308 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:33.170078 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.171128 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.172149 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.172491 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.172585 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.174927 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:33.174938 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:33.174948 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:33.174969 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:33.176383 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.176846 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.176851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.176853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.176944 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.177239 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.177242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.177244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.177495 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:33.177616 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.177964 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.178123 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":117,"threads_started":1}
I20260809 04:38:33.179184 11686 tablet_service.cc:1514] Processing DeleteTablet for tablet a413eb2e976d4d06a8f8234a1b6f32cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:38:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:47606
I20260809 04:38:33.179991 11792 tablet_replica.cc:330] stopping tablet replica
I20260809 04:38:33.180262 11792 raft_consensus.cc:2229] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:38:33.180492 11792 raft_consensus.cc:2258] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:38:33.181445 11792 ts_tablet_manager.cc:1905] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:38:33.184885 11792 ts_tablet_manager.cc:1918] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:38:33.184983 11792 log.cc:1196] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal/wals/a413eb2e976d4d06a8f8234a1b6f32cb
I20260809 04:38:33.185266 11792 ts_tablet_manager.cc:1939] T a413eb2e976d4d06a8f8234a1b6f32cb P 9409bf8060a343788fa2e514a2935cc0: Deleting consensus metadata
I20260809 04:38:33.187002 11488 catalog_manager.cc:4841] TS 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977): tablet a413eb2e976d4d06a8f8234a1b6f32cb (table test-table [id=8a19b451a714489ab146a0f360a133c0]) successfully deleted
W20260809 04:38:33.270350 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:33.270650 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:33.270713 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:33.275068 11785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:33.275133 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:33.275213 11785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:38:33.280838 11785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:36937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=42807
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:33.281960 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:33.282985 11785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:38:33.292774 11802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:33.292816 11785 server_base.cc:1034] running on GCE node
W20260809 04:38:33.292779 11804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:33.292774 11806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:33.293509 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:33.294181 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:33.295331 11785 hybrid_clock.cc:648] HybridClock initialized: now 1786250313295300 us; error 43 us; skew 500 ppm
Aug 09 04:38:33 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:33.298113 11785 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:38:33.299041 11785 webserver.cc:460] Webserver started at http://127.19.92.2:42807/ using document root <none> and password file <none>
I20260809 04:38:33.299582 11785 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:33.299666 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:33.303686 11785 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:38:33.305951 11816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:33.307304 11785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:38:33.307431 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "2acc874aad744fe399f748363809cde8"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f723354b1cebd840acb1da3112b6b712"
server_key_iv: "0e7ad4b30f2383ee7c966ef493763785"
server_key_version: "encryptionkey@0"
I20260809 04:38:33.308043 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:33.330785 11785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:33.334328 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:33.334586 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:33.335256 11785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:33.336210 11785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:38:33.336261 11785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:33.336319 11785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:38:33.336347 11785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:33.346109 11785 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:36937
I20260809 04:38:33.346166 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:36937 every 8 connection(s)
I20260809 04:38:33.347294 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:38:33.348373 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 11785
I20260809 04:38:33.348829 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:43081
--local_ip_for_outbound_sockets=127.19.92.3
--tserver_master_addrs=127.19.92.62:41319
--webserver_port=33855
--webserver_interface=127.19.92.3
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:38:33 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:38:33.361924 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.349516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:43547 (local address 127.19.92.62:41319)
0809 04:38:33.349655 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.349659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.351237 (+ 1578us) server_negotiation.cc:407] Connection header received
0809 04:38:33.352403 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.352405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.352463 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.352539 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:33.353829 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.354371 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.355574 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.355810 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.355905 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.358489 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:33.358493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:33.358495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:33.358531 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:33.359850 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.360350 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.360353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.360354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.360433 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.360750 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.360754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.360755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.360945 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:33.361038 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.361605 (+ 567us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.361750 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:38:33.362764 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.350139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:43547)
0809 04:38:33.351038 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.351099 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:33.352168 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:33.352700 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:33.352708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:33.353095 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:33.353657 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.353672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.354505 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.354508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.355429 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.355440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.355963 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.355967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.356147 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.356778 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:33.356795 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:33.358295 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:33.359962 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:33.359968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:33.359985 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:33.360250 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:33.360516 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:33.360519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:33.360521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:33.360662 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:33.361050 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:33.361058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:33.361370 (+ 312us) client_negotiation.cc:769] Sending connection context
0809 04:38:33.361552 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 04:38:33.361893 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":176,"threads_started":1}
I20260809 04:38:33.364018 11981 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41319
I20260809 04:38:33.364264 11981 heartbeater.cc:464] Registering TS with master...
I20260809 04:38:33.364784 11981 heartbeater.cc:511] Master 127.19.92.62:41319 requested a full tablet report, sending...
I20260809 04:38:33.365886 11508 ts_manager.cc:194] Registered new tserver with Master: 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937)
I20260809 04:38:33.366461 11508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:43547
W20260809 04:38:33.466480 11985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:33.466760 11985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:33.466821 11985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:33.470909 11985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:33.470980 11985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:33.471060 11985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:38:33.476563 11985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:43081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=33855
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:33.477692 11985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:33.478760 11985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:38:33.488306 12002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:33.488313 11995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:33.488371 11985 server_base.cc:1034] running on GCE node
W20260809 04:38:33.488313 11999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:33.488950 11985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:33.489640 11985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:33.490800 11985 hybrid_clock.cc:648] HybridClock initialized: now 1786250313490770 us; error 40 us; skew 500 ppm
Aug 09 04:38:33 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:33.493557 11985 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:38:33.494441 11985 webserver.cc:460] Webserver started at http://127.19.92.3:33855/ using document root <none> and password file <none>
I20260809 04:38:33.495011 11985 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:33.495086 11985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:33.499080 11985 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:38:33.501274 12016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:33.502530 11985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:38:33.502638 11985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "7560ad4155a64a8cb492e0b2c476e7e4"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f2ee41c5a86dad74c9c8bd3b66f493ac"
server_key_iv: "362e48687d753e0f0e7e9a79b1f3d6ad"
server_key_version: "encryptionkey@0"
I20260809 04:38:33.503189 11985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:33.519461 11985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:33.522789 11985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:33.522950 11985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:33.523535 11985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:33.524371 11985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:38:33.524437 11985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:33.524485 11985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:38:33.524504 11985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:33.534114 11985 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:43081
I20260809 04:38:33.534269 12172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:43081 every 8 connection(s)
I20260809 04:38:33.535181 11985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:38:33.537688 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 11985
Aug 09 04:38:33 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250313, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:38:33.546552 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.538339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.19.92.62:41319)
0809 04:38:33.538576 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.538580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.538785 (+ 205us) server_negotiation.cc:407] Connection header received
0809 04:38:33.538911 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.538914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.538961 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.539040 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:33.539991 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.540563 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.541612 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.541841 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.541929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.543334 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:33.543337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:33.543339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:33.543364 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:33.544713 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.545225 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.545229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.545231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.545339 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.545670 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.545672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.545673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.545859 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:33.546036 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.546241 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.546373 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
I20260809 04:38:33.549832 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.537074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:41877 (local address 127.19.92.62:41319)
0809 04:38:33.537250 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.537253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.538450 (+ 1197us) server_negotiation.cc:407] Connection header received
0809 04:38:33.539406 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.539410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.539455 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.539546 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:33.541038 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.541559 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.542901 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.543123 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.543213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.546064 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:33.546067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:33.546070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:33.546095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:33.547642 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.548210 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.548213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.548215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.548295 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.548648 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.548651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.548653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.548845 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:33.548934 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.549552 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.549676 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:38:33.551021 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.537527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:41877)
0809 04:38:33.538289 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.538345 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:33.539222 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:33.539883 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:33.539893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:33.540282 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:33.540902 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.540922 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.541688 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.541691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.542769 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.542781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.543272 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.543276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.543458 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.543998 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:33.544020 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:33.545870 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:33.547774 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:33.547782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:33.547799 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:33.548111 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:33.548383 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:33.548386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:33.548388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:33.548552 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:33.548961 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:33.548968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:33.549301 (+ 333us) client_negotiation.cc:769] Sending connection context
0809 04:38:33.549499 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 04:38:33.549883 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":132,"threads_started":1}
I20260809 04:38:33.552212 12173 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41319
I20260809 04:38:33.552495 12173 heartbeater.cc:464] Registering TS with master...
I20260809 04:38:33.553114 12173 heartbeater.cc:511] Master 127.19.92.62:41319 requested a full tablet report, sending...
I20260809 04:38:33.554399 11508 ts_manager.cc:194] Registered new tserver with Master: 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081)
I20260809 04:38:33.555020 11508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:41877
I20260809 04:38:33.560179 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:38:33.564100 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.560788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.19.92.62:41319)
0809 04:38:33.560917 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.560920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.561097 (+ 177us) server_negotiation.cc:407] Connection header received
0809 04:38:33.561326 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.561329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.561439 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.561541 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:38:33.561745 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.562255 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.563067 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.563276 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.563344 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.563616 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:38:33.563754 (+ 138us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 04:38:33.563989 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59918: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 04:38:33.564168 11780 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59918: token expired
W20260809 04:38:33.564527 11409 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.92.62:41319)
I20260809 04:38:33.571578 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.564877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.19.92.62:41319)
0809 04:38:33.565001 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.565004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.565017 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:38:33.565065 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.565067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.565105 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.565221 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:33.565836 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.566322 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.567121 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.567325 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.567403 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.568656 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:33.568659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:33.568661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:33.568684 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:33.570129 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.570559 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.570562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.570563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.570645 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:33.570880 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:33.570883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:33.570884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:33.571077 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:33.571184 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.571363 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.571459 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:38:33.574230 11508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:38:33.577610 11508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:38:33.586056 11686 tablet_service.cc:1467] Processing CreateTablet for tablet c54e9ac5262044dbb497721109fa23cb (DEFAULT_TABLE table=test-table [id=0173509430a04b5297b0abca2884d895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:38:33.586809 11686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54e9ac5262044dbb497721109fa23cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:38:33.589723 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.584879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43081 (local address 127.0.0.1:45558)
0809 04:38:33.585236 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.585250 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:33.585343 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:33.586104 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:33.586107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:33.586126 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:33.586224 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.586231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.587328 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.587332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.588366 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.588377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.589332 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.589335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.589422 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.589483 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 04:38:33.589527 (+ 44us) client_negotiation.cc:240] Negotiation successful
0809 04:38:33.589599 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":32,"thread_start_us":115,"threads_started":1}
I20260809 04:38:33.590058 11627 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Bootstrap starting.
I20260809 04:38:33.590466 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.585304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.19.92.3:43081)
0809 04:38:33.585628 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.585637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.585657 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:38:33.585739 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.585744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.585969 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.586121 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:33.586306 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.587207 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.588502 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.589187 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.590025 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.590184 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.590242 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.590348 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":119,"threads_started":1}
I20260809 04:38:33.590948 11627 tablet_bootstrap.cc:654] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Neither blocks nor log segments found. Creating new log.
I20260809 04:38:33.591939 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.584374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:36937 (local address 127.0.0.1:52810)
0809 04:38:33.584537 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.584554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:33.584639 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:33.585739 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:33.585742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:33.585763 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:33.585883 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.585889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.587445 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.587448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.588563 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.588574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.591566 (+ 2992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.591570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.591668 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.591693 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:38:33.591742 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 04:38:33.591821 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":3}
I20260809 04:38:33.592456 11627 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: No bootstrap required, opened a new log
I20260809 04:38:33.592551 11627 ts_tablet_manager.cc:1397] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:38:33.592641 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.584596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.19.92.2:36937)
0809 04:38:33.585117 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.585134 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.585199 (+ 65us) server_negotiation.cc:407] Connection header received
0809 04:38:33.585291 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.585297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.585567 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.585756 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:33.585990 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.587119 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.588880 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.589875 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.592219 (+ 2344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.592373 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.592418 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.592490 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":129,"threads_started":1}
I20260809 04:38:33.593035 11627 raft_consensus.cc:348] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.593180 11627 raft_consensus.cc:374] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:38:33.593236 11627 raft_consensus.cc:729] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9409bf8060a343788fa2e514a2935cc0, State: Initialized, Role: FOLLOWER
I20260809 04:38:33.593362 11627 consensus_queue.cc:260] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.593626 11627 ts_tablet_manager.cc:1428] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 04:38:33.593943 12081 tablet_service.cc:1467] Processing CreateTablet for tablet c54e9ac5262044dbb497721109fa23cb (DEFAULT_TABLE table=test-table [id=0173509430a04b5297b0abca2884d895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:38:33.595160 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54e9ac5262044dbb497721109fa23cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:38:33.595323 11889 tablet_service.cc:1467] Processing CreateTablet for tablet c54e9ac5262044dbb497721109fa23cb (DEFAULT_TABLE table=test-table [id=0173509430a04b5297b0abca2884d895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:38:33.596283 11889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54e9ac5262044dbb497721109fa23cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:38:33.602200 12188 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: Bootstrap starting.
I20260809 04:38:33.602906 12189 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: Bootstrap starting.
I20260809 04:38:33.604596 12188 tablet_bootstrap.cc:654] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: Neither blocks nor log segments found. Creating new log.
I20260809 04:38:33.605443 12189 tablet_bootstrap.cc:654] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: Neither blocks nor log segments found. Creating new log.
I20260809 04:38:33.605960 12188 log.cc:824] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:33.606268 12189 log.cc:824] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:33.607803 12188 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: No bootstrap required, opened a new log
I20260809 04:38:33.607955 12189 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: No bootstrap required, opened a new log
I20260809 04:38:33.607983 12188 ts_tablet_manager.cc:1397] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:38:33.608141 12189 ts_tablet_manager.cc:1397] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 04:38:33.611148 12188 raft_consensus.cc:348] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.611424 12188 raft_consensus.cc:374] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:38:33.611482 12188 raft_consensus.cc:729] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7560ad4155a64a8cb492e0b2c476e7e4, State: Initialized, Role: FOLLOWER
I20260809 04:38:33.611315 12189 raft_consensus.cc:348] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.611614 12189 raft_consensus.cc:374] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:38:33.611670 12189 raft_consensus.cc:729] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2acc874aad744fe399f748363809cde8, State: Initialized, Role: FOLLOWER
I20260809 04:38:33.611950 12188 consensus_queue.cc:260] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.612037 12189 consensus_queue.cc:260] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.612639 12173 heartbeater.cc:503] Master 127.19.92.62:41319 was elected leader, sending a full tablet report...
I20260809 04:38:33.612954 12188 ts_tablet_manager.cc:1428] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:38:33.613058 12189 ts_tablet_manager.cc:1428] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260809 04:38:33.613742 11981 heartbeater.cc:503] Master 127.19.92.62:41319 was elected leader, sending a full tablet report...
I20260809 04:38:33.614276 12198 raft_consensus.cc:482] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:38:33.614408 12198 raft_consensus.cc:504] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.615509 12198 leader_election.cc:288] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081), 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977)
I20260809 04:38:33.619665 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.615852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:57625 (local address 127.19.92.3:43081)
0809 04:38:33.615960 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.615964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.616435 (+ 471us) server_negotiation.cc:407] Connection header received
0809 04:38:33.616528 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.616530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.616571 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.616647 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:33.616972 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.617737 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.618773 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.619168 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.619258 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.619321 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.619483 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.619551 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:38:33.619685 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.615769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43081 (local address 127.19.92.2:57625)
0809 04:38:33.616348 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.616360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:33.616457 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:33.616671 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:33.616674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:33.616692 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:33.616820 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.616826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.617837 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.617840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.618656 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.618665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.619269 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.619272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.619372 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.619396 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:38:33.619469 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 04:38:33.619534 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":215,"thread_start_us":132,"threads_started":1}
I20260809 04:38:33.619916 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.615767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45977 (local address 127.19.92.2:49107)
0809 04:38:33.616261 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.616275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:33.616361 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:33.617067 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:33.617070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:33.617090 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:38:33.617215 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.617222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.618170 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.618173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.618884 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:33.618892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.619551 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.619555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:33.619649 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.619668 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:38:33.619726 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 04:38:33.619800 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424}
I20260809 04:38:33.619917 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.616675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:49107 (local address 127.19.92.1:45977)
0809 04:38:33.616796 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.616800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.616812 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:38:33.616861 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.616864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.616929 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.617015 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:38:33.617362 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.618069 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.619009 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.619435 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.619535 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.619691 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.619737 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.619801 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:38:33.620493 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54e9ac5262044dbb497721109fa23cb" candidate_uuid: "2acc874aad744fe399f748363809cde8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" is_pre_election: true
I20260809 04:38:33.620482 11718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54e9ac5262044dbb497721109fa23cb" candidate_uuid: "2acc874aad744fe399f748363809cde8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9409bf8060a343788fa2e514a2935cc0" is_pre_election: true
I20260809 04:38:33.620772 12109 raft_consensus.cc:2454] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2acc874aad744fe399f748363809cde8 in term 0.
I20260809 04:38:33.620772 11718 raft_consensus.cc:2454] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2acc874aad744fe399f748363809cde8 in term 0.
I20260809 04:38:33.621268 11820 leader_election.cc:302] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4; no voters:
I20260809 04:38:33.621532 12198 raft_consensus.cc:2781] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:38:33.621626 12198 raft_consensus.cc:482] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:38:33.621670 12198 raft_consensus.cc:3037] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:38:33.622671 12198 raft_consensus.cc:504] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.622967 12198 leader_election.cc:288] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [CANDIDATE]: Term 1 election: Requested vote from peers 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081), 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977)
I20260809 04:38:33.623311 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54e9ac5262044dbb497721109fa23cb" candidate_uuid: "2acc874aad744fe399f748363809cde8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7560ad4155a64a8cb492e0b2c476e7e4"
I20260809 04:38:33.623306 11718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54e9ac5262044dbb497721109fa23cb" candidate_uuid: "2acc874aad744fe399f748363809cde8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9409bf8060a343788fa2e514a2935cc0"
I20260809 04:38:33.623411 12109 raft_consensus.cc:3037] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:38:33.623421 11718 raft_consensus.cc:3037] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:38:33.624387 12109 raft_consensus.cc:2454] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2acc874aad744fe399f748363809cde8 in term 1.
I20260809 04:38:33.624567 11718 raft_consensus.cc:2454] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2acc874aad744fe399f748363809cde8 in term 1.
I20260809 04:38:33.624698 11820 leader_election.cc:302] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4; no voters:
I20260809 04:38:33.624871 12198 raft_consensus.cc:2781] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:38:33.625140 12198 raft_consensus.cc:686] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 1 LEADER]: Becoming Leader. State: Replica: 2acc874aad744fe399f748363809cde8, State: Running, Role: LEADER
I20260809 04:38:33.625581 12198 consensus_queue.cc:237] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:33.628875 11508 catalog_manager.cc:5495] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2acc874aad744fe399f748363809cde8 (127.19.92.2). New cstate: current_term: 1 leader_uuid: "2acc874aad744fe399f748363809cde8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } health_report { overall_health: UNKNOWN } } }
I20260809 04:38:33.641077 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.637892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.19.92.2:36937)
0809 04:38:33.638024 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:33.638027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:33.638040 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:38:33.638176 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:33.638179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:33.638234 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:33.638318 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:38:33.638496 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.638994 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.639808 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:33.640040 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:33.640117 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:33.640351 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:38:33.640765 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:38:33.640854 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:33.640882 (+ 28us) server_negotiation.cc:299] Negotiation successful
0809 04:38:33.640949 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 04:38:33.641426 11778 tablet.cc:2369] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:38:33.646783 11718 raft_consensus.cc:1261] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Refusing update from remote peer 2acc874aad744fe399f748363809cde8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:38:33.646793 12109 raft_consensus.cc:1261] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 1 FOLLOWER]: Refusing update from remote peer 2acc874aad744fe399f748363809cde8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:38:33.647476 12202 consensus_queue.cc:1035] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:38:33.647650 12198 consensus_queue.cc:1035] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:38:33.654222 12207 mvcc.cc:205] Tried to move back new op lower bound from 7316481284690300928 to 7316481284612960256. Current Snapshot: MvccSnapshot[applied={T|T < 7316481284690300928}]
I20260809 04:38:33.655393 12205 mvcc.cc:205] Tried to move back new op lower bound from 7316481284690300928 to 7316481284612960256. Current Snapshot: MvccSnapshot[applied={T|T < 7316481284690300928}]
I20260809 04:38:33.664722 11508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:38:33.664899 11508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:38:33.667582 11508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Sending DeleteTablet for 3 replicas of tablet c54e9ac5262044dbb497721109fa23cb
I20260809 04:38:33.668352 11686 tablet_service.cc:1514] Processing DeleteTablet for tablet c54e9ac5262044dbb497721109fa23cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:38:33 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:47606
I20260809 04:38:33.668447 11889 tablet_service.cc:1514] Processing DeleteTablet for tablet c54e9ac5262044dbb497721109fa23cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:38:33 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:52810
I20260809 04:38:33.668445 12081 tablet_service.cc:1514] Processing DeleteTablet for tablet c54e9ac5262044dbb497721109fa23cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:38:33 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:45558
I20260809 04:38:33.668596 11792 tablet_replica.cc:330] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: stopping tablet replica
I20260809 04:38:33.668713 11792 raft_consensus.cc:2229] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:38:33.668874 11792 raft_consensus.cc:2258] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:38:33.669077 12218 tablet_replica.cc:330] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: stopping tablet replica
I20260809 04:38:33.669101 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 11559
I20260809 04:38:33.669468 12218 raft_consensus.cc:2229] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:38:33.669796 12218 raft_consensus.cc:2258] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:38:33.674369 12219 tablet_replica.cc:330] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: stopping tablet replica
I20260809 04:38:33.674713 12219 raft_consensus.cc:2229] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:38:33.675132 12219 raft_consensus.cc:2258] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:38:33.676090 12218 ts_tablet_manager.cc:1905] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:38:33.678014 12219 ts_tablet_manager.cc:1905] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:38:33.680462 12218 ts_tablet_manager.cc:1918] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:38:33.680706 12218 log.cc:1196] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/c54e9ac5262044dbb497721109fa23cb
I20260809 04:38:33.680927 12219 ts_tablet_manager.cc:1918] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:38:33.681044 12219 log.cc:1196] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/c54e9ac5262044dbb497721109fa23cb
I20260809 04:38:33.681294 12218 ts_tablet_manager.cc:1939] T c54e9ac5262044dbb497721109fa23cb P 7560ad4155a64a8cb492e0b2c476e7e4: Deleting consensus metadata
I20260809 04:38:33.681416 12219 ts_tablet_manager.cc:1939] T c54e9ac5262044dbb497721109fa23cb P 2acc874aad744fe399f748363809cde8: Deleting consensus metadata
I20260809 04:38:33.682750 11487 catalog_manager.cc:4841] TS 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937): tablet c54e9ac5262044dbb497721109fa23cb (table test-table [id=0173509430a04b5297b0abca2884d895]) successfully deleted
I20260809 04:38:33.683274 11490 catalog_manager.cc:4841] TS 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081): tablet c54e9ac5262044dbb497721109fa23cb (table test-table [id=0173509430a04b5297b0abca2884d895]) successfully deleted
W20260809 04:38:33.689014 11488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:45977 (error 108)
I20260809 04:38:33.689256 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 11785
I20260809 04:38:33.695712 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:33.695340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:45977 (local address 127.0.0.1:47616)
0809 04:38:33.695511 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:33.695606 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 04:38:33.695930 11488 catalog_manager.cc:4584] TS 9409bf8060a343788fa2e514a2935cc0 (127.19.92.1:45977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c54e9ac5262044dbb497721109fa23cb: Network error: Client connection negotiation failed: client connection to 127.19.92.1:45977: connect: Connection refused (error 111)
I20260809 04:38:33.700637 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 11985
I20260809 04:38:33.715276 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 11462
I20260809 04:38:53.731950 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41319
--webserver_interface=127.19.92.62
--webserver_port=41705
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:38:53.850929 12222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:53.851200 12222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:53.851258 12222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:53.855212 12222 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 04:38:53.855263 12222 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 04:38:53.855285 12222 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 04:38:53.855302 12222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:38:53.855327 12222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:53.855361 12222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:53.855388 12222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:38:53.855418 12222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:38:53.860610 12222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:53.861696 12222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:53.862727 12222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:38:53.871547 12222 server_base.cc:1034] running on GCE node
W20260809 04:38:53.871585 12231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:53.871469 12234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:53.871469 12229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:53.872393 12222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:53.873634 12222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:53.874800 12222 hybrid_clock.cc:648] HybridClock initialized: now 1786250333874763 us; error 52 us; skew 500 ppm
Aug 09 04:38:53 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250333, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:53.883785 12222 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:38:53.884909 12222 webserver.cc:460] Webserver started at http://127.19.92.62:41705/ using document root <none> and password file <none>
I20260809 04:38:53.885557 12222 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:53.885634 12222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:53.889778 12222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:38:53.891973 12244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:53.893239 12222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:38:53.893356 12222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "92e181e571a24d809eb7abf7e6c56875"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "89045383d6804a5ca6a813b90597cc15"
server_key_iv: "013f35c31926559fbbc7dc82ab51356e"
server_key_version: "encryptionkey@0"
I20260809 04:38:53.893903 12222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:53.907851 12222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:53.911109 12222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:53.911298 12222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:53.918787 12222 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41319
I20260809 04:38:53.919083 12316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41319 every 8 connection(s)
I20260809 04:38:53.919939 12222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:38:53.920389 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 12222
I20260809 04:38:53.920846 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:45977
--local_ip_for_outbound_sockets=127.19.92.1
--tserver_master_addrs=127.19.92.62:41319
--webserver_port=35551
--webserver_interface=127.19.92.1
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:38:53.924481 12317 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:38:53.926287 12317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Bootstrap starting.
I20260809 04:38:53.943292 12317 log.cc:824] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:53.954869 12317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:38:53.955284 12317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Bootstrap complete.
I20260809 04:38:53.959306 12317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:53.959736 12317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e181e571a24d809eb7abf7e6c56875, State: Initialized, Role: FOLLOWER
I20260809 04:38:53.960233 12317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:53.960353 12317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:38:53.960419 12317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:38:53.960523 12317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 2 FOLLOWER]: Advancing to term 3
I20260809 04:38:53.962491 12317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:53.962823 12317 leader_election.cc:302] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e181e571a24d809eb7abf7e6c56875; no voters:
I20260809 04:38:53.963135 12317 leader_election.cc:288] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 04:38:53.963244 12321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 04:38:53.963457 12321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [term 3 LEADER]: Becoming Leader. State: Replica: 92e181e571a24d809eb7abf7e6c56875, State: Running, Role: LEADER
I20260809 04:38:53.963768 12321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } }
I20260809 04:38:53.964519 12317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:38:53.965252 12322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "92e181e571a24d809eb7abf7e6c56875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } } }
I20260809 04:38:53.965389 12322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: This master's current role is: LEADER
I20260809 04:38:53.965682 12323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e181e571a24d809eb7abf7e6c56875. Latest consensus state: current_term: 3 leader_uuid: "92e181e571a24d809eb7abf7e6c56875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e181e571a24d809eb7abf7e6c56875" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41319 } } }
I20260809 04:38:53.965767 12323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875 [sys.catalog]: This master's current role is: LEADER
I20260809 04:38:53.966256 12328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:38:53.971403 12328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:38:53.972191 12328 catalog_manager.cc:1259] Loaded cluster ID: 1b9363a75ab5494882b8bbcc4e4ea6c8
I20260809 04:38:53.972254 12328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:38:53.974817 12328 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:38:53.975572 12328 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Loaded TSK: 0
I20260809 04:38:53.975657 12328 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92e181e571a24d809eb7abf7e6c56875: Loaded TSK: 1
F20260809 04:38:53.977036 12328 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f43cee8a9ad google::LogMessage::Fail() at ??:0
@ 0x7f43cee8d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f43cee8a4de google::LogMessage::Flush() at ??:0
@ 0x7f43cee8ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f43d1b4e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f43d1b46f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f43d97ef7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f43d97c999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f43d97ca178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f43d980bdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f43d982a82b std::function<>::operator()() at ??:0
@ 0x7f43d97c9b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f43d97caf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f43d97c7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f43d980b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f43d10dc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f43d10dcc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f43d10de3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f43d10cd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f43d27116db start_thread at ??:0
@ 0x7f43cd98861f clone at ??:0
W20260809 04:38:54.061408 12319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:54.061770 12319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:54.061851 12319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:54.067845 12319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:54.067932 12319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:54.068023 12319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:38:54.076045 12319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:45977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=35551
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:54.077713 12319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:54.079149 12319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:38:54.090838 12386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:54.090852 12389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:54.090838 12387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:54.091693 12319 server_base.cc:1034] running on GCE node
I20260809 04:38:54.092207 12319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:54.092869 12319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:54.094036 12319 hybrid_clock.cc:648] HybridClock initialized: now 1786250334094013 us; error 38 us; skew 500 ppm
Aug 09 04:38:54 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250334, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:54.096844 12319 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:38:54.097813 12319 webserver.cc:460] Webserver started at http://127.19.92.1:35551/ using document root <none> and password file <none>
I20260809 04:38:54.098451 12319 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:54.098528 12319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:54.102654 12319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:38:54.105118 12399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:54.106514 12319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:38:54.106642 12319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "9409bf8060a343788fa2e514a2935cc0"
format_stamp: "Formatted at 2026-08-09 04:38:11 on dist-test-slave-rmhg"
server_key: "5f017bc88f01d950154b00f3c36aab22"
server_key_iv: "d6633e58c7fcc2f1c870410773812d12"
server_key_version: "encryptionkey@0"
I20260809 04:38:54.107185 12319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:54.121877 12319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:54.125370 12319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:54.125573 12319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:54.126252 12319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:54.127429 12409 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 04:38:54.133174 12319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 04:38:54.133257 12319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260809 04:38:54.133339 12319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 04:38:54.135105 12319 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 04:38:54.135155 12319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 04:38:54.135484 12409 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Bootstrap starting.
I20260809 04:38:54.145349 12319 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:45977
I20260809 04:38:54.145609 12569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:45977 every 8 connection(s)
I20260809 04:38:54.146397 12319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:38:54.150224 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 12319
I20260809 04:38:54.150679 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:36937
--local_ip_for_outbound_sockets=127.19.92.2
--tserver_master_addrs=127.19.92.62:41319
--webserver_port=42807
--webserver_interface=127.19.92.2
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:38:54.160301 12409 log.cc:824] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Log is configured to *not* fsync() on all Append() calls
I20260809 04:38:54.167783 12409 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:38:54.168155 12409 tablet_bootstrap.cc:492] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Bootstrap complete.
I20260809 04:38:54.169070 12409 ts_tablet_manager.cc:1397] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.008s
I20260809 04:38:54.171558 12409 raft_consensus.cc:348] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:54.171979 12409 raft_consensus.cc:729] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9409bf8060a343788fa2e514a2935cc0, State: Initialized, Role: FOLLOWER
I20260809 04:38:54.172451 12409 consensus_queue.cc:260] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:54.173437 12409 ts_tablet_manager.cc:1428] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0: Time spent starting tablet: real 0.004s user 0.007s sys 0.001s
W20260809 04:38:54.239701 12573 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:38:54.148371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:38897)
0809 04:38:54.148886 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.148941 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:54.149799 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:54.238848 (+ 89049us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
W20260809 04:38:54.240338 12570 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:41319 (0 consecutive failures): Network error: Failed to ping master at 127.19.92.62:41319: Client connection negotiation failed: client connection to 127.19.92.62:41319: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 04:38:54.241238 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:54.240899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:35031)
0809 04:38:54.241055 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.241143 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 04:38:54.242017 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:54.241772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:44343)
0809 04:38:54.241905 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.241956 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 04:38:54.242192 12570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:38:54.294866 12574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:54.295135 12574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:54.295176 12574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:54.299196 12574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:54.299252 12574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:54.299315 12574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:38:54.304600 12574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:36937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=42807
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:54.305750 12574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:54.306811 12574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:38:54.316756 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:54.316787 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:54.316787 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:54.317130 12574 server_base.cc:1034] running on GCE node
I20260809 04:38:54.317612 12574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:54.318322 12574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:54.319486 12574 hybrid_clock.cc:648] HybridClock initialized: now 1786250334319459 us; error 45 us; skew 500 ppm
Aug 09 04:38:54 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250334, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:54.322248 12574 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:38:54.323208 12574 webserver.cc:460] Webserver started at http://127.19.92.2:42807/ using document root <none> and password file <none>
I20260809 04:38:54.323753 12574 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:54.323815 12574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:54.327983 12574 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:38:54.330189 12599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:54.331533 12574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:38:54.331653 12574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "2acc874aad744fe399f748363809cde8"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f723354b1cebd840acb1da3112b6b712"
server_key_iv: "0e7ad4b30f2383ee7c966ef493763785"
server_key_version: "encryptionkey@0"
I20260809 04:38:54.332209 12574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:54.342406 12574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:54.345412 12574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:54.345590 12574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:54.346204 12574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:54.347045 12574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:38:54.347091 12574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:54.347134 12574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:38:54.347163 12574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:54.356237 12574 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:36937
I20260809 04:38:54.356289 12770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:36937 every 8 connection(s)
I20260809 04:38:54.357425 12574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:38:54.361379 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:54.359727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:58663)
0809 04:38:54.360438 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.360599 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":126,"threads_started":1}
W20260809 04:38:54.361920 12771 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:41319 (0 consecutive failures): Network error: Failed to ping master at 127.19.92.62:41319: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
I20260809 04:38:54.362556 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:54.362306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:43113)
0809 04:38:54.362438 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.362500 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 04:38:54.363197 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:54.362953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:36665)
0809 04:38:54.363071 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.363118 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 04:38:54.363382 12771 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 04:38:54.363508 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 12574
I20260809 04:38:54.363938 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:43081
--local_ip_for_outbound_sockets=127.19.92.3
--tserver_master_addrs=127.19.92.62:41319
--webserver_port=33855
--webserver_interface=127.19.92.3
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:38:54.480064 12775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:38:54.480370 12775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:38:54.480414 12775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:38:54.484509 12775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:38:54.484573 12775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:38:54.484643 12775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:38:54.489961 12775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:43081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=33855
--tserver_master_addrs=127.19.92.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:38:54.491005 12775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:38:54.492039 12775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:38:54.501484 12775 server_base.cc:1034] running on GCE node
W20260809 04:38:54.501437 12785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:54.501468 12788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:38:54.501443 12783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:38:54.502045 12775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:38:54.502700 12775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:38:54.503865 12775 hybrid_clock.cc:648] HybridClock initialized: now 1786250334503845 us; error 30 us; skew 500 ppm
Aug 09 04:38:54 dist-test-slave-rmhg krb5kdc[10658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250334, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:38:54.506596 12775 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:38:54.507467 12775 webserver.cc:460] Webserver started at http://127.19.92.3:33855/ using document root <none> and password file <none>
I20260809 04:38:54.508019 12775 fs_manager.cc:359] Metadata directory not provided
I20260809 04:38:54.508088 12775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:38:54.512142 12775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:38:54.514528 12802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:54.516013 12775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:38:54.516151 12775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "7560ad4155a64a8cb492e0b2c476e7e4"
format_stamp: "Formatted at 2026-08-09 04:38:12 on dist-test-slave-rmhg"
server_key: "f2ee41c5a86dad74c9c8bd3b66f493ac"
server_key_iv: "362e48687d753e0f0e7e9a79b1f3d6ad"
server_key_version: "encryptionkey@0"
I20260809 04:38:54.516778 12775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:38:54.536567 12775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:38:54.539608 12775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:38:54.539801 12775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:38:54.540453 12775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:38:54.541371 12775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:38:54.541430 12775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:54.541487 12775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:38:54.541514 12775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:38:54.550221 12775 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:43081
I20260809 04:38:54.550253 12981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:43081 every 8 connection(s)
I20260809 04:38:54.551249 12775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:38:54.551551 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 12775
I20260809 04:38:54.555104 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:54.553482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:55343)
0809 04:38:54.554194 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.554309 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":120,"threads_started":1}
W20260809 04:38:54.555709 12982 heartbeater.cc:644] Failed to heartbeat to 127.19.92.62:41319 (0 consecutive failures): Network error: Failed to ping master at 127.19.92.62:41319: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
I20260809 04:38:54.556540 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:54.556145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:37979)
0809 04:38:54.556309 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.556378 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 04:38:54.557320 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:54.557087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:43337)
0809 04:38:54.557222 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:54.557268 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 04:38:54.557518 12982 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 04:38:55.243419 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:55.242912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:48513)
0809 04:38:55.243231 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:55.243301 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260809 04:38:55.364439 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:55.364009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:46077)
0809 04:38:55.364261 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:55.364322 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260809 04:38:55.558630 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:55.558137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:54245)
0809 04:38:55.558426 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:55.558516 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":78,"threads_started":1}
I20260809 04:38:55.734519 12991 raft_consensus.cc:482] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:38:55.734730 12991 raft_consensus.cc:504] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:55.736091 12991 leader_election.cc:288] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937), 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081)
Aug 09 04:38:55 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250334, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.2@KRBTEST.COM
Aug 09 04:38:55 dist-test-slave-rmhg krb5kdc[10658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250334, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.3@KRBTEST.COM
I20260809 04:38:55.750934 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:55.736365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:36937 (local address 127.19.92.1:59877)
0809 04:38:55.737088 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:55.737098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:55.737231 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:55.739419 (+ 2188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:55.739421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:55.739596 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:55.739880 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:55.739895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:55.741287 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:55.741292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:55.742854 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:55.742866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:55.743616 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:55.743619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:55.743772 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:55.744429 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:55.744449 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:55.746167 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:55.748316 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:55.748323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:55.748338 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:55.748718 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:55.749060 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:55.749063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:55.749064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:55.749274 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:55.749912 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:55.749918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:55.750196 (+ 278us) client_negotiation.cc:769] Sending connection context
0809 04:38:55.750362 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 04:38:55.750732 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":594,"spinlock_wait_cycles":11264,"thread_start_us":80,"threads_started":1}
I20260809 04:38:55.750962 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:55.736640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:54153 (local address 127.19.92.3:43081)
0809 04:38:55.737045 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:55.737068 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:55.737112 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:38:55.738078 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:55.738087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:55.738558 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:55.738903 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:55.740059 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:55.741202 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:55.743794 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:55.744436 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:55.744574 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:55.746638 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:55.746643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:55.746654 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:55.746685 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:55.748332 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:55.748798 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:55.748805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:55.748811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:55.748933 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:55.749352 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:55.749355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:55.749357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:55.749795 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:55.749917 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:55.750503 (+ 586us) server_negotiation.cc:299] Negotiation successful
0809 04:38:55.750770 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":103,"threads_started":1}
I20260809 04:38:55.750959 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:55.736504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:59877 (local address 127.19.92.2:36937)
0809 04:38:55.736866 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0809 04:38:55.736891 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 04:38:55.737211 (+ 320us) server_negotiation.cc:407] Connection header received
0809 04:38:55.738078 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:38:55.738088 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:38:55.738558 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:38:55.738903 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:38:55.740054 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:55.741112 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:55.742996 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:38:55.743503 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:38:55.743641 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:55.746444 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:38:55.746452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:38:55.746471 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:38:55.746520 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:38:55.748181 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:55.748835 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:55.748840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:55.748844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:55.748963 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:38:55.749366 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:38:55.749369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:38:55.749370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:38:55.749794 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:38:55.749917 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 04:38:55.750517 (+ 600us) server_negotiation.cc:299] Negotiation successful
0809 04:38:55.750767 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260809 04:38:55.750957 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:55.736365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43081 (local address 127.19.92.1:54153)
0809 04:38:55.736525 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:55.736538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:38:55.736638 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:38:55.738980 (+ 2342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:38:55.738985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:38:55.739366 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:38:55.739881 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:55.739897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:55.741381 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:55.741384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:55.742854 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:38:55.742866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:38:55.744559 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:38:55.744562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:38:55.744652 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:38:55.745056 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:38:55.745072 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:38:55.746444 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:38:55.748447 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:55.748450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:55.748451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:55.748685 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:55.749051 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:38:55.749054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:38:55.749056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:38:55.749264 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:38:55.749912 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:38:55.749918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:38:55.750199 (+ 281us) client_negotiation.cc:769] Sending connection context
0809 04:38:55.750362 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 04:38:55.750811 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":339,"spinlock_wait_cycles":23808}
W20260809 04:38:55.754247 12404 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
W20260809 04:38:55.754434 12400 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
I20260809 04:38:55.754527 12400 leader_election.cc:302] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9409bf8060a343788fa2e514a2935cc0; no voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4
I20260809 04:38:55.754832 12991 raft_consensus.cc:2726] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:38:56.244657 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:56.244290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:57363)
0809 04:38:56.244505 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:56.244570 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 04:38:56.365731 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:56.365256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:45743)
0809 04:38:56.365570 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:56.365634 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260809 04:38:56.559893 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:56.559416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:49809)
0809 04:38:56.559731 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:56.559795 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
I20260809 04:38:57.246023 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:57.245546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:41345)
0809 04:38:57.245842 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:57.245910 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
I20260809 04:38:57.366878 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:57.366470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:37339)
0809 04:38:57.366707 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:57.366765 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260809 04:38:57.561267 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:57.560765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:46533)
0809 04:38:57.561056 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:57.561122 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
I20260809 04:38:57.719668 13005 raft_consensus.cc:482] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:38:57.719801 13005 raft_consensus.cc:504] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:38:57.720258 13005 leader_election.cc:288] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937), 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081)
W20260809 04:38:57.721186 12400 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
W20260809 04:38:57.721308 12404 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
I20260809 04:38:57.721411 12404 leader_election.cc:302] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9409bf8060a343788fa2e514a2935cc0; no voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4
I20260809 04:38:57.721628 13005 raft_consensus.cc:2726] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:38:58.247337 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:58.246859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:52827)
0809 04:38:58.247168 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:58.247236 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260809 04:38:58.368068 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:58.367615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:54201)
0809 04:38:58.367896 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:58.367970 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260809 04:38:58.562592 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:58.562077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:58617)
0809 04:38:58.562409 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:58.562491 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260809 04:38:59.248912 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:59.248388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:56315)
0809 04:38:59.248745 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:59.248811 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":78,"threads_started":1}
I20260809 04:38:59.369380 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:59.368840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:35745)
0809 04:38:59.369138 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:59.369248 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260809 04:38:59.563913 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:38:59.563420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:34499)
0809 04:38:59.563748 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0809 04:38:59.563811 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":121,"threads_started":1}
I20260809 04:39:00.250171 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:00.249771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:47731)
0809 04:39:00.250016 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:00.250071 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260809 04:39:00.370590 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:00.370171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:40653)
0809 04:39:00.370432 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:00.370493 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260809 04:39:00.525379 13020 raft_consensus.cc:482] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:00.525520 13020 raft_consensus.cc:504] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:39:00.525940 13020 leader_election.cc:288] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937), 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081)
W20260809 04:39:00.526729 12400 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
W20260809 04:39:00.526844 12404 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
I20260809 04:39:00.526924 12404 leader_election.cc:302] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9409bf8060a343788fa2e514a2935cc0; no voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4
I20260809 04:39:00.527065 13020 raft_consensus.cc:2726] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:39:00.565304 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:00.564807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:59231)
0809 04:39:00.565102 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:00.565195 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260809 04:39:01.251447 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:01.250992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:46057)
0809 04:39:01.251286 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:01.251350 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260809 04:39:01.371758 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:01.371294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:34841)
0809 04:39:01.371603 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:01.371662 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":107,"threads_started":1}
I20260809 04:39:01.566828 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:01.566295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:46337)
0809 04:39:01.566639 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:01.566704 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":92,"threads_started":1}
I20260809 04:39:02.252738 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:02.252281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:51579)
0809 04:39:02.252576 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:02.252642 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260809 04:39:02.372918 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:02.372469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:47893)
0809 04:39:02.372762 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:02.372819 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260809 04:39:02.568181 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:02.567705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:56309)
0809 04:39:02.568012 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:02.568080 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260809 04:39:03.254073 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:03.253628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:37023)
0809 04:39:03.253901 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:03.253969 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260809 04:39:03.374207 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:03.373746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:36641)
0809 04:39:03.374044 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:03.374107 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":86,"threads_started":1}
I20260809 04:39:03.527591 13036 raft_consensus.cc:482] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:03.527729 13036 raft_consensus.cc:504] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:39:03.528146 13036 leader_election.cc:288] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937), 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081)
W20260809 04:39:03.528903 12400 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
W20260809 04:39:03.529029 12404 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
I20260809 04:39:03.529116 12404 leader_election.cc:302] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9409bf8060a343788fa2e514a2935cc0; no voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4
I20260809 04:39:03.529301 13036 raft_consensus.cc:2726] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:39:03.569402 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:03.568964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:33059)
0809 04:39:03.569245 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:03.569304 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260809 04:39:04.255545 13038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:04.255051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:53973)
0809 04:39:04.255356 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:04.255423 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
I20260809 04:39:04.375576 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:04.375080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:43791)
0809 04:39:04.375353 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:04.375423 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260809 04:39:04.570678 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:04.570193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:58319)
0809 04:39:04.570515 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:04.570581 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260809 04:39:05.256898 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:05.256390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:51143)
0809 04:39:05.256734 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:05.256799 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260809 04:39:05.376801 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:05.376359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:60083)
0809 04:39:05.376643 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:05.376703 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260809 04:39:05.571851 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:05.571433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:38661)
0809 04:39:05.571697 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:05.571755 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260809 04:39:06.258141 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:06.257661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:41293)
0809 04:39:06.257978 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:06.258043 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":82,"threads_started":1}
I20260809 04:39:06.378069 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:06.377625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:49443)
0809 04:39:06.377913 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:06.377968 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260809 04:39:06.573242 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:06.572721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:59149)
0809 04:39:06.573045 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:06.573114 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260809 04:39:07.259328 13050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:07.258872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:38059)
0809 04:39:07.259165 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:07.259225 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260809 04:39:07.379388 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:07.378903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:41885)
0809 04:39:07.379198 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:07.379266 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":106,"threads_started":1}
I20260809 04:39:07.574591 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:07.574099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:35321)
0809 04:39:07.574415 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:07.574490 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260809 04:39:08.260628 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:08.260179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:46601)
0809 04:39:08.260466 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:08.260529 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
I20260809 04:39:08.381229 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:08.380422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:54337)
0809 04:39:08.381014 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:08.381077 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":96,"threads_started":1}
I20260809 04:39:08.575884 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:08.575456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:55841)
0809 04:39:08.575722 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:08.575786 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260809 04:39:09.034971 13058 raft_consensus.cc:482] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:09.035122 13058 raft_consensus.cc:504] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2acc874aad744fe399f748363809cde8" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 36937 } } peers { permanent_uuid: "7560ad4155a64a8cb492e0b2c476e7e4" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43081 } } peers { permanent_uuid: "9409bf8060a343788fa2e514a2935cc0" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 45977 } }
I20260809 04:39:09.035549 13058 leader_election.cc:288] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937), 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081)
W20260809 04:39:09.036371 12400 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2acc874aad744fe399f748363809cde8 (127.19.92.2:36937): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
W20260809 04:39:09.036479 12404 leader_election.cc:341] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7560ad4155a64a8cb492e0b2c476e7e4 (127.19.92.3:43081): Not found: Tablet not found: c54e9ac5262044dbb497721109fa23cb
I20260809 04:39:09.036586 12404 leader_election.cc:302] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9409bf8060a343788fa2e514a2935cc0; no voters: 2acc874aad744fe399f748363809cde8, 7560ad4155a64a8cb492e0b2c476e7e4
I20260809 04:39:09.036731 13058 raft_consensus.cc:2726] T c54e9ac5262044dbb497721109fa23cb P 9409bf8060a343788fa2e514a2935cc0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:39:09.262066 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:09.261602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.1:60557)
0809 04:39:09.261901 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:09.261962 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
I20260809 04:39:09.382582 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:09.382124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.2:48207)
0809 04:39:09.382394 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:09.382454 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260809 04:39:09.556042 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 12319
I20260809 04:39:09.577443 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:09.576862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41319 (local address 127.19.92.3:36149)
0809 04:39:09.577252 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:09.577319 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.62:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":148,"threads_started":1}
I20260809 04:39:09.577770 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 12574
I20260809 04:39:09.587729 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 12775
2026-08-09T04:39:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59340 ms)
[----------] 30 tests from SecurityITest (252543 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:09 dist-test-slave-rmhg krb5kdc[13067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:09 dist-test-slave-rmhg krb5kdc[13067](info): set up 2 sockets
Aug 09 04:39:09 dist-test-slave-rmhg krb5kdc[13067](info): commencing operation
krb5kdc: starting...
W20260809 04:39:10.725288 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:10 dist-test-slave-rmhg krb5kdc[13067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:10.856227 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:43627
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:43627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:10.969218 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:10.969511 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:10.969552 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:10.973567 13083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:10.973623 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:10.973640 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:10.973655 13083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:10.973668 13083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:10.979156 13083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:43627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:43627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:10.980176 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:10.981165 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:39:10.989713 13083 server_base.cc:1034] running on GCE node
W20260809 04:39:10.989835 13099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:10.989835 13093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:10.989917 13094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:10.990648 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:10.991763 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:10.992926 13083 hybrid_clock.cc:648] HybridClock initialized: now 1786250350992910 us; error 30 us; skew 500 ppm
Aug 09 04:39:10 dist-test-slave-rmhg krb5kdc[13067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250350, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:10.995642 13083 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:10.996539 13083 webserver.cc:460] Webserver started at http://127.19.92.62:36091/ using document root <none> and password file <none>
I20260809 04:39:10.997083 13083 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:10.997145 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:10.997411 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:10.999558 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "7b3cde0398ed40efb70b9a018e530fed"
format_stamp: "Formatted at 2026-08-09 04:39:10 on dist-test-slave-rmhg"
server_key: "8a17f008fab015031d8b983d0e2b117b"
server_key_iv: "3fcdfe1e804bdc3574e32420b2918b98"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.000244 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "7b3cde0398ed40efb70b9a018e530fed"
format_stamp: "Formatted at 2026-08-09 04:39:10 on dist-test-slave-rmhg"
server_key: "8a17f008fab015031d8b983d0e2b117b"
server_key_iv: "3fcdfe1e804bdc3574e32420b2918b98"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.004523 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 04:39:11.007186 13113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.008651 13083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:39:11.008811 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "7b3cde0398ed40efb70b9a018e530fed"
format_stamp: "Formatted at 2026-08-09 04:39:10 on dist-test-slave-rmhg"
server_key: "8a17f008fab015031d8b983d0e2b117b"
server_key_iv: "3fcdfe1e804bdc3574e32420b2918b98"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.008951 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:11.029098 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:11.032191 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:11.032379 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:11.039634 13083 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:43627
I20260809 04:39:11.039674 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:43627 every 8 connection(s)
I20260809 04:39:11.040681 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:11.043309 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 13083
I20260809 04:39:11.043445 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:11.043672 13194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:11.043753 19824 external_mini_cluster.cc:1428] Setting key a03dda22d09a3f2937a1b21724013b51
I20260809 04:39:11.051287 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed: Bootstrap starting.
Aug 09 04:39:11 dist-test-slave-rmhg krb5kdc[13067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:11.053658 13194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:11.054517 13194 log.cc:824] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:11.056816 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed: No bootstrap required, opened a new log
I20260809 04:39:11.058626 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.045248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.19.92.62:43627)
0809 04:39:11.045709 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:11.045743 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:11.045788 (+ 45us) server_negotiation.cc:407] Connection header received
0809 04:39:11.046622 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:11.046648 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:11.047102 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:11.047504 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:11.048232 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.049763 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.050660 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.051340 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.051509 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.053983 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:11.053989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:11.054007 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:11.054040 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:11.055820 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.056326 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.056330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.056335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.056441 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.056740 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.056742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.056743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.057181 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:11.057339 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:11.057617 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 04:39:11.057885 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260809 04:39:11.060927 13194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3cde0398ed40efb70b9a018e530fed" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43627 } }
I20260809 04:39:11.061177 13194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:11.061237 13194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3cde0398ed40efb70b9a018e530fed, State: Initialized, Role: FOLLOWER
I20260809 04:39:11.061648 13194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3cde0398ed40efb70b9a018e530fed" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43627 } }
I20260809 04:39:11.061791 13194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:11.061854 13194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:11.061957 13194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:11.062844 13194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3cde0398ed40efb70b9a018e530fed" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43627 } }
I20260809 04:39:11.063187 13194 leader_election.cc:302] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b3cde0398ed40efb70b9a018e530fed; no voters:
I20260809 04:39:11.063511 13194 leader_election.cc:288] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:11.063629 13199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:11.063841 13199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [term 1 LEADER]: Becoming Leader. State: Replica: 7b3cde0398ed40efb70b9a018e530fed, State: Running, Role: LEADER
I20260809 04:39:11.064194 13199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3cde0398ed40efb70b9a018e530fed" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43627 } }
I20260809 04:39:11.064723 13194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:11.065766 13201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b3cde0398ed40efb70b9a018e530fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3cde0398ed40efb70b9a018e530fed" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43627 } } }
I20260809 04:39:11.065891 13201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:11.066033 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b3cde0398ed40efb70b9a018e530fed. Latest consensus state: current_term: 1 leader_uuid: "7b3cde0398ed40efb70b9a018e530fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3cde0398ed40efb70b9a018e530fed" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43627 } } }
I20260809 04:39:11.066219 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed [sys.catalog]: This master's current role is: LEADER
W20260809 04:39:11.069686 13221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:11.069762 13221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:11.069839 13220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:11.070802 13220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:11.076531 13220 catalog_manager.cc:1347] Generated new cluster ID: 7d5b9a4c3dc04ccdb180fbb10784f2a9
I20260809 04:39:11.076609 13220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:11.093370 13220 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:11.094367 13220 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:11.117308 13220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed: Generated new TSK 0
I20260809 04:39:11.118184 13220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:39:11.150277 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43627
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:11.287767 13227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:11.288046 13227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:11.288089 13227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:11.292241 13227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:11.292316 13227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:11.292375 13227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:39:11.297881 13227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:11.298933 13227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:11.299955 13227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:11.310034 13240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:11.310034 13232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:11.310041 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:11.310094 13227 server_base.cc:1034] running on GCE node
I20260809 04:39:11.310683 13227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:11.311395 13227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:11.312587 13227 hybrid_clock.cc:648] HybridClock initialized: now 1786250351312565 us; error 33 us; skew 500 ppm
Aug 09 04:39:11 dist-test-slave-rmhg krb5kdc[13067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:11.315637 13227 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:39:11.316620 13227 webserver.cc:460] Webserver started at http://127.19.92.1:46185/ using document root <none> and password file <none>
I20260809 04:39:11.317320 13227 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:11.317401 13227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:11.317623 13227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:11.319825 13227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "87e477eb455c4f9fbcb28fd4f1832315"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "b3466a89b05f9aaf5fd66e70f9992462"
server_key_iv: "f4d3bdc54b4a179044b4338f83b2423a"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.320470 13227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "87e477eb455c4f9fbcb28fd4f1832315"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "b3466a89b05f9aaf5fd66e70f9992462"
server_key_iv: "f4d3bdc54b4a179044b4338f83b2423a"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.324859 13227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:39:11.327797 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.329392 13227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:11.329535 13227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "87e477eb455c4f9fbcb28fd4f1832315"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "b3466a89b05f9aaf5fd66e70f9992462"
server_key_iv: "f4d3bdc54b4a179044b4338f83b2423a"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.329654 13227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:11.344926 13227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:11.347996 13227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:11.348168 13227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:11.348737 13227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:11.349653 13227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:11.349702 13227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.349743 13227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:11.349763 13227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.358345 13227 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:39235
I20260809 04:39:11.358376 13418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:39235 every 8 connection(s)
I20260809 04:39:11.359341 13227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:39:11.368984 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 13227
I20260809 04:39:11.369118 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:39:11.369463 19824 external_mini_cluster.cc:1428] Setting key 996c40a39a75b08575fc445ad3b30e48
Aug 09 04:39:11 dist-test-slave-rmhg krb5kdc[13067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:11.374364 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.361455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:43221 (local address 127.19.92.62:43627)
0809 04:39:11.361615 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:11.361620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:11.362688 (+ 1068us) server_negotiation.cc:407] Connection header received
0809 04:39:11.364040 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:11.364043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:11.364097 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:11.364180 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:11.365632 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.366217 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.367431 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.367691 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.367773 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.370483 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:11.370487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:11.370490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:11.370517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:11.371961 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.372596 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.372600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.372601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.372684 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.373067 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.373071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.373073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.373365 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:11.373506 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:11.374047 (+ 541us) server_negotiation.cc:299] Negotiation successful
0809 04:39:11.374189 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:39:11.375229 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.361937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43627 (local address 127.19.92.1:43221)
0809 04:39:11.362512 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:11.362563 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:11.363792 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:11.364459 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:11.364468 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:11.364876 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:11.365482 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.365500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.366337 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.366340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.367307 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.367322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.367837 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.367840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.367996 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.368588 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:11.368605 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:11.370325 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:11.372084 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:11.372090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:11.372109 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:11.372471 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:11.372766 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:11.372768 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:11.372770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:11.372961 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:11.373525 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:11.373531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:11.373812 (+ 281us) client_negotiation.cc:769] Sending connection context
0809 04:39:11.373981 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 04:39:11.374263 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":124,"threads_started":1}
I20260809 04:39:11.376528 13419 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43627
I20260809 04:39:11.376758 13419 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:11.377285 13419 heartbeater.cc:511] Master 127.19.92.62:43627 requested a full tablet report, sending...
I20260809 04:39:11.379020 13137 ts_manager.cc:194] Registered new tserver with Master: 87e477eb455c4f9fbcb28fd4f1832315 (127.19.92.1:39235)
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:39:11.380640 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:43221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:39:11.398109 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43627
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:11.513295 13430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:11.513598 13430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:11.513661 13430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:11.517863 13430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:11.517936 13430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:11.518019 13430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:39:11.523746 13430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:11.525007 13430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:11.526155 13430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:11.536118 13437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:11.536098 13440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:11.536130 13435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:11.536325 13430 server_base.cc:1034] running on GCE node
I20260809 04:39:11.536798 13430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:11.537451 13430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:11.538609 13430 hybrid_clock.cc:648] HybridClock initialized: now 1786250351538589 us; error 30 us; skew 500 ppm
Aug 09 04:39:11 dist-test-slave-rmhg krb5kdc[13067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:11.541296 13430 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:39:11.542173 13430 webserver.cc:460] Webserver started at http://127.19.92.2:45371/ using document root <none> and password file <none>
I20260809 04:39:11.542713 13430 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:11.542781 13430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:11.542996 13430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:11.545090 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "6a75a808c7d44033b3cd1c5fadefe988"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "780b5d48f08be93b9f42c3e0be9734c2"
server_key_iv: "45fad69b9f7fe85e0501a2e0f19a2059"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.545673 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "6a75a808c7d44033b3cd1c5fadefe988"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "780b5d48f08be93b9f42c3e0be9734c2"
server_key_iv: "45fad69b9f7fe85e0501a2e0f19a2059"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.549379 13430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:39:11.551785 13448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.552961 13430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:11.553083 13430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "6a75a808c7d44033b3cd1c5fadefe988"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "780b5d48f08be93b9f42c3e0be9734c2"
server_key_iv: "45fad69b9f7fe85e0501a2e0f19a2059"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.553213 13430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:11.566789 13430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:11.569759 13430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:11.569947 13430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:11.570562 13430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:11.571460 13430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:11.571511 13430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.571563 13430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:11.571588 13430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.579934 13430 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:45719
I20260809 04:39:11.579979 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:45719 every 8 connection(s)
I20260809 04:39:11.580868 13430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:39:11.585776 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 13430
I20260809 04:39:11.585886 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:39:11.586176 19824 external_mini_cluster.cc:1428] Setting key 52217762daa1c311b568e9ca94bd1ee8
Aug 09 04:39:11 dist-test-slave-rmhg krb5kdc[13067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:11.594554 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.582724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:39019 (local address 127.19.92.62:43627)
0809 04:39:11.582890 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:11.582895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:11.583986 (+ 1091us) server_negotiation.cc:407] Connection header received
0809 04:39:11.584835 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:11.584837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:11.584888 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:11.584976 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:11.586309 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.586973 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.588220 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.588518 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.588624 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.591202 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:11.591205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:11.591207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:11.591232 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:11.592483 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.592964 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.592968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.592970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.593049 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.593402 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.593405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.593407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.593604 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:11.593694 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:11.594231 (+ 537us) server_negotiation.cc:299] Negotiation successful
0809 04:39:11.594366 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:39:11.595360 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.583153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43627 (local address 127.19.92.2:39019)
0809 04:39:11.583829 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:11.583871 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:11.584656 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:11.585130 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:11.585137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:11.585605 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:11.586149 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.586163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.587101 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.587104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.588100 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.588110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.588652 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.588655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.588799 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.589391 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:11.589408 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:11.591030 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:11.592591 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:11.592597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:11.592614 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:11.592871 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:11.593131 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:11.593134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:11.593136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:11.593314 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:11.593706 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:11.593714 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:11.593989 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 04:39:11.594156 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 04:39:11.594496 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:39:11.596591 13630 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43627
I20260809 04:39:11.596850 13630 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:11.597440 13630 heartbeater.cc:511] Master 127.19.92.62:43627 requested a full tablet report, sending...
I20260809 04:39:11.598554 13137 ts_manager.cc:194] Registered new tserver with Master: 6a75a808c7d44033b3cd1c5fadefe988 (127.19.92.2:45719)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:39:11.599200 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:39019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:39:11.615618 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43627
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:11.735649 13638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:11.735925 13638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:11.735973 13638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:11.740164 13638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:11.740252 13638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:11.740312 13638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:39:11.746022 13638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:11.747138 13638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:11.748170 13638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:11.758255 13649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:11.758285 13647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:11.758262 13652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:11.758702 13638 server_base.cc:1034] running on GCE node
I20260809 04:39:11.759166 13638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:11.759871 13638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:11.761075 13638 hybrid_clock.cc:648] HybridClock initialized: now 1786250351761055 us; error 35 us; skew 500 ppm
Aug 09 04:39:11 dist-test-slave-rmhg krb5kdc[13067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:11.764276 13638 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:39:11.765498 13638 webserver.cc:460] Webserver started at http://127.19.92.3:42711/ using document root <none> and password file <none>
I20260809 04:39:11.766183 13638 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:11.766259 13638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:11.766484 13638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:11.768779 13638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "b557682c925d4f8eb2378377a09c51f8"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "b7d395f04a4df310f34c74c28ed50051"
server_key_iv: "9adb2d221acc0f05e586c3fccd83b967"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.769431 13638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "b557682c925d4f8eb2378377a09c51f8"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "b7d395f04a4df310f34c74c28ed50051"
server_key_iv: "9adb2d221acc0f05e586c3fccd83b967"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.773674 13638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:39:11.776394 13661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.777814 13638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:11.777948 13638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "b557682c925d4f8eb2378377a09c51f8"
format_stamp: "Formatted at 2026-08-09 04:39:11 on dist-test-slave-rmhg"
server_key: "b7d395f04a4df310f34c74c28ed50051"
server_key_iv: "9adb2d221acc0f05e586c3fccd83b967"
server_key_version: "encryptionkey@0"
I20260809 04:39:11.778069 13638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:11.801064 13638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:11.804644 13638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:11.804859 13638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:11.805573 13638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:11.806484 13638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:11.806545 13638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.806600 13638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:11.806628 13638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:11.815943 13638 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:43147
I20260809 04:39:11.815972 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:43147 every 8 connection(s)
I20260809 04:39:11.816896 13638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:39:11.824167 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 13638
I20260809 04:39:11.824334 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:39:11.824687 19824 external_mini_cluster.cc:1428] Setting key 9df9bfda6067d93ad9665ee8a4ff2a7b
Aug 09 04:39:11 dist-test-slave-rmhg krb5kdc[13067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250351, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:11.831415 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.819031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:38159 (local address 127.19.92.62:43627)
0809 04:39:11.819181 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:11.819186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:11.820088 (+ 902us) server_negotiation.cc:407] Connection header received
0809 04:39:11.821503 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:11.821507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:11.821569 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:11.821650 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:11.822933 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.823480 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.825092 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.825417 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.825512 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.828142 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:11.828146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:11.828148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:11.828173 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:11.829615 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.830112 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.830115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.830116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.830198 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.830455 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.830457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.830458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.830630 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:11.830703 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:11.831186 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 04:39:11.831290 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:39:11.832298 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.819419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43627 (local address 127.19.92.3:38159)
0809 04:39:11.819925 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:11.819972 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:11.821277 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:11.821847 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:11.821854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:11.822240 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:11.822778 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.822793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.823607 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.823610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.824935 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.824948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.825569 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.825574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.825771 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.826405 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:11.826423 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:11.827979 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:11.829725 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:11.829730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:11.829744 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:11.830033 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:11.830264 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:11.830266 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:11.830267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:11.830393 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:11.830711 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:11.830716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:11.830988 (+ 272us) client_negotiation.cc:769] Sending connection context
0809 04:39:11.831147 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 04:39:11.831410 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":127,"threads_started":1}
I20260809 04:39:11.833513 13825 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43627
I20260809 04:39:11.833796 13825 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:11.834327 13825 heartbeater.cc:511] Master 127.19.92.62:43627 requested a full tablet report, sending...
I20260809 04:39:11.835348 13137 ts_manager.cc:194] Registered new tserver with Master: b557682c925d4f8eb2378377a09c51f8 (127.19.92.3:43147)
I20260809 04:39:11.835907 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:38159
I20260809 04:39:11.838776 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:39:11.848173 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.840804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.19.92.62:43627)
0809 04:39:11.840955 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:11.840960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:11.841207 (+ 247us) server_negotiation.cc:407] Connection header received
0809 04:39:11.841338 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:11.841340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:11.841391 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:11.841460 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:11.842192 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.842743 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.843590 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.843819 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.843923 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.845263 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:11.845266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:11.845268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:11.845294 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:11.846627 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.847022 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.847025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.847026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.847116 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:11.847353 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:11.847355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:11.847356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:11.847584 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:11.847689 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:11.847877 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 04:39:11.848001 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 04:39:11.851204 13137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:39:11.854298 13137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:39:11.868793 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.862621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.19.92.2:45719)
0809 04:39:11.863870 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:11.863881 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:11.863901 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:39:11.863981 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:11.863986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:11.864152 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:11.864289 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:11.864712 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.866058 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.867495 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.868138 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.868236 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.868385 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:11.868550 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 04:39:11.868638 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":116,"threads_started":1}
I20260809 04:39:11.868783 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.862472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:45719 (local address 127.0.0.1:40832)
0809 04:39:11.863128 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:11.863178 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:11.863302 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:11.864301 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:11.864307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:11.864347 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:11.864581 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.864589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.866226 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.866229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.867363 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.867377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.868257 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.868261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.868363 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.868399 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 04:39:11.868511 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 04:39:11.868633 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":227,"spinlock_wait_cycles":6912,"thread_start_us":67,"threads_started":1}
I20260809 04:39:11.869102 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.863292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.19.92.3:43147)
0809 04:39:11.863629 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:11.863651 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:11.863674 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:39:11.863757 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:11.863762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:11.863976 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:11.864135 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:11.864766 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.865662 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.867413 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.868191 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.868299 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.868437 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:11.868861 (+ 424us) server_negotiation.cc:299] Negotiation successful
0809 04:39:11.868956 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":142,"threads_started":1}
I20260809 04:39:11.869266 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.862560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43147 (local address 127.0.0.1:51576)
0809 04:39:11.863325 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:11.863338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:11.863403 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:11.864310 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:11.864313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:11.864349 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:11.864629 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.864637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.865788 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.865791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.867281 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.867292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.868516 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.868520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.868623 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.868646 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:39:11.868711 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 04:39:11.868782 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":107,"spinlock_wait_cycles":4864,"thread_start_us":84,"threads_started":1}
I20260809 04:39:11.869464 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.863261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.19.92.1:39235)
0809 04:39:11.863731 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:11.863748 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:11.863960 (+ 212us) server_negotiation.cc:407] Connection header received
0809 04:39:11.864052 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:11.864058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:11.864260 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:11.864474 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:11.864732 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.865983 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.867703 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.868474 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.868568 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.868695 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:11.869235 (+ 540us) server_negotiation.cc:299] Negotiation successful
0809 04:39:11.869331 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":91,"threads_started":1}
I20260809 04:39:11.869107 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:11.862778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:39235 (local address 127.0.0.1:36398)
0809 04:39:11.863561 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:11.863572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:11.863647 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:11.864485 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:11.864487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:11.864503 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:11.864648 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.864653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.866128 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.866131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.867592 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:11.867601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:11.868749 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:11.868752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:11.868843 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:11.868864 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:39:11.868925 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 04:39:11.868980 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":221,"thread_start_us":99,"threads_started":1}
I20260809 04:39:11.872040 13326 tablet_service.cc:1467] Processing CreateTablet for tablet b645f8b12b0c47228ad0b40f674bb35d (DEFAULT_TABLE table=test-table [id=5463484cdf2b49f683ebba0911e7f9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:11.872174 13728 tablet_service.cc:1467] Processing CreateTablet for tablet b645f8b12b0c47228ad0b40f674bb35d (DEFAULT_TABLE table=test-table [id=5463484cdf2b49f683ebba0911e7f9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:11.872174 13519 tablet_service.cc:1467] Processing CreateTablet for tablet b645f8b12b0c47228ad0b40f674bb35d (DEFAULT_TABLE table=test-table [id=5463484cdf2b49f683ebba0911e7f9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:11.873018 13326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b645f8b12b0c47228ad0b40f674bb35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:11.873214 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b645f8b12b0c47228ad0b40f674bb35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:11.873206 13519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b645f8b12b0c47228ad0b40f674bb35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:11.879391 13852 tablet_bootstrap.cc:492] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315: Bootstrap starting.
I20260809 04:39:11.881035 13856 tablet_bootstrap.cc:492] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Bootstrap starting.
I20260809 04:39:11.881708 13852 tablet_bootstrap.cc:654] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:11.882513 13852 log.cc:824] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:11.883263 13855 tablet_bootstrap.cc:492] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Bootstrap starting.
I20260809 04:39:11.883496 13856 tablet_bootstrap.cc:654] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:11.884495 13856 log.cc:824] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:11.884775 13852 tablet_bootstrap.cc:492] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315: No bootstrap required, opened a new log
I20260809 04:39:11.885016 13852 ts_tablet_manager.cc:1397] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:39:11.885635 13855 tablet_bootstrap.cc:654] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:11.886421 13856 tablet_bootstrap.cc:492] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: No bootstrap required, opened a new log
I20260809 04:39:11.886605 13856 ts_tablet_manager.cc:1397] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:39:11.886873 13855 log.cc:824] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:11.888903 13855 tablet_bootstrap.cc:492] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: No bootstrap required, opened a new log
I20260809 04:39:11.889101 13855 ts_tablet_manager.cc:1397] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:39:11.889441 13852 raft_consensus.cc:348] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:11.889564 13856 raft_consensus.cc:348] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:11.889734 13852 raft_consensus.cc:374] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:11.889786 13852 raft_consensus.cc:729] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e477eb455c4f9fbcb28fd4f1832315, State: Initialized, Role: FOLLOWER
I20260809 04:39:11.889816 13856 raft_consensus.cc:374] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:11.889895 13856 raft_consensus.cc:729] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a75a808c7d44033b3cd1c5fadefe988, State: Initialized, Role: FOLLOWER
I20260809 04:39:11.890298 13856 consensus_queue.cc:260] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:11.890285 13852 consensus_queue.cc:260] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:11.891001 13419 heartbeater.cc:503] Master 127.19.92.62:43627 was elected leader, sending a full tablet report...
I20260809 04:39:11.891317 13852 ts_tablet_manager.cc:1428] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:39:11.892342 13856 ts_tablet_manager.cc:1428] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260809 04:39:11.892717 13630 heartbeater.cc:503] Master 127.19.92.62:43627 was elected leader, sending a full tablet report...
I20260809 04:39:11.892612 13855 raft_consensus.cc:348] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:11.892902 13855 raft_consensus.cc:374] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:11.892952 13855 raft_consensus.cc:729] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b557682c925d4f8eb2378377a09c51f8, State: Initialized, Role: FOLLOWER
I20260809 04:39:11.893421 13855 consensus_queue.cc:260] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:11.894096 13825 heartbeater.cc:503] Master 127.19.92.62:43627 was elected leader, sending a full tablet report...
I20260809 04:39:11.894384 13855 ts_tablet_manager.cc:1428] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260809 04:39:12.067994 13826 tablet.cc:2369] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:39:12.082048 13632 tablet.cc:2369] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:39:12.110844 13421 tablet.cc:2369] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:12.315411 13862 raft_consensus.cc:482] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:12.315672 13862 raft_consensus.cc:504] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:12.316872 13862 leader_election.cc:288] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a75a808c7d44033b3cd1c5fadefe988 (127.19.92.2:45719), b557682c925d4f8eb2378377a09c51f8 (127.19.92.3:43147)
I20260809 04:39:12.322278 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.317272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:56127 (local address 127.19.92.2:45719)
0809 04:39:12.317403 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:12.317407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:12.317638 (+ 231us) server_negotiation.cc:407] Connection header received
0809 04:39:12.317811 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:12.317815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:12.317870 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:12.317946 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:12.318345 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.319161 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.320853 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.321520 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.321650 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:12.321710 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:12.322061 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 04:39:12.322139 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:39:12.322278 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.317324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:35689 (local address 127.19.92.3:43147)
0809 04:39:12.317452 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:12.317456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:12.317621 (+ 165us) server_negotiation.cc:407] Connection header received
0809 04:39:12.317820 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:12.317823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:12.317869 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:12.317945 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:12.318345 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.319161 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.320854 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.321539 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.321639 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:12.321709 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:12.322061 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 04:39:12.322139 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:39:12.322346 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.317113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:45719 (local address 127.19.92.1:56127)
0809 04:39:12.317544 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:12.317555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:12.317701 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:12.317963 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:12.317966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:12.317991 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:12.318231 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:12.318239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.319284 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.319288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:12.320720 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:12.320733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.321667 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.321672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:12.321867 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:12.321939 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 04:39:12.322026 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 04:39:12.322136 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":233,"spinlock_wait_cycles":17152,"thread_start_us":134,"threads_started":1}
I20260809 04:39:12.322346 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.317207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:43147 (local address 127.19.92.1:35689)
0809 04:39:12.317525 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:12.317539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:12.317701 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:12.317963 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:12.317966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:12.317991 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:12.318231 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:12.318239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.319284 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.319288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:12.320720 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:12.320733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.321667 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.321672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:12.321892 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:12.321944 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 04:39:12.322022 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:39:12.322136 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":158,"spinlock_wait_cycles":69120,"thread_start_us":121,"threads_started":1}
I20260809 04:39:12.323047 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b645f8b12b0c47228ad0b40f674bb35d" candidate_uuid: "87e477eb455c4f9fbcb28fd4f1832315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a75a808c7d44033b3cd1c5fadefe988" is_pre_election: true
I20260809 04:39:12.323223 13761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b645f8b12b0c47228ad0b40f674bb35d" candidate_uuid: "87e477eb455c4f9fbcb28fd4f1832315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b557682c925d4f8eb2378377a09c51f8" is_pre_election: true
I20260809 04:39:12.323390 13552 raft_consensus.cc:2454] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e477eb455c4f9fbcb28fd4f1832315 in term 0.
I20260809 04:39:12.323534 13761 raft_consensus.cc:2454] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e477eb455c4f9fbcb28fd4f1832315 in term 0.
I20260809 04:39:12.323948 13252 leader_election.cc:302] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a75a808c7d44033b3cd1c5fadefe988, 87e477eb455c4f9fbcb28fd4f1832315; no voters:
I20260809 04:39:12.324259 13862 raft_consensus.cc:2781] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:39:12.324354 13862 raft_consensus.cc:482] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:39:12.324402 13862 raft_consensus.cc:3037] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:12.325563 13862 raft_consensus.cc:504] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:12.325902 13862 leader_election.cc:288] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [CANDIDATE]: Term 1 election: Requested vote from peers 6a75a808c7d44033b3cd1c5fadefe988 (127.19.92.2:45719), b557682c925d4f8eb2378377a09c51f8 (127.19.92.3:43147)
I20260809 04:39:12.326305 13761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b645f8b12b0c47228ad0b40f674bb35d" candidate_uuid: "87e477eb455c4f9fbcb28fd4f1832315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b557682c925d4f8eb2378377a09c51f8"
I20260809 04:39:12.326306 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b645f8b12b0c47228ad0b40f674bb35d" candidate_uuid: "87e477eb455c4f9fbcb28fd4f1832315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a75a808c7d44033b3cd1c5fadefe988"
I20260809 04:39:12.326442 13552 raft_consensus.cc:3037] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:12.326442 13761 raft_consensus.cc:3037] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:12.327740 13761 raft_consensus.cc:2454] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e477eb455c4f9fbcb28fd4f1832315 in term 1.
I20260809 04:39:12.327741 13552 raft_consensus.cc:2454] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e477eb455c4f9fbcb28fd4f1832315 in term 1.
I20260809 04:39:12.328053 13252 leader_election.cc:302] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a75a808c7d44033b3cd1c5fadefe988, 87e477eb455c4f9fbcb28fd4f1832315; no voters:
I20260809 04:39:12.328209 13862 raft_consensus.cc:2781] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:12.328437 13862 raft_consensus.cc:686] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 1 LEADER]: Becoming Leader. State: Replica: 87e477eb455c4f9fbcb28fd4f1832315, State: Running, Role: LEADER
I20260809 04:39:12.328802 13862 consensus_queue.cc:237] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } }
I20260809 04:39:12.331996 13137 catalog_manager.cc:5495] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e477eb455c4f9fbcb28fd4f1832315 (127.19.92.1). New cstate: current_term: 1 leader_uuid: "87e477eb455c4f9fbcb28fd4f1832315" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87e477eb455c4f9fbcb28fd4f1832315" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39235 } health_report { overall_health: HEALTHY } } }
I20260809 04:39:12.430001 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.426184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.19.92.1:39235)
0809 04:39:12.426400 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:12.426404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:12.426419 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:39:12.426582 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:12.426585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:12.426645 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:12.426747 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:12.426937 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.427454 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.428313 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.428571 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.428659 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:12.429064 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:12.429630 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:12.429734 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:12.429810 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 04:39:12.429865 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260809 04:39:12.435809 13552 raft_consensus.cc:1261] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 1 FOLLOWER]: Refusing update from remote peer 87e477eb455c4f9fbcb28fd4f1832315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:12.435808 13761 raft_consensus.cc:1261] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 1 FOLLOWER]: Refusing update from remote peer 87e477eb455c4f9fbcb28fd4f1832315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:12.436535 13868 consensus_queue.cc:1035] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a75a808c7d44033b3cd1c5fadefe988" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:12.436719 13862 consensus_queue.cc:1035] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b557682c925d4f8eb2378377a09c51f8" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 43147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:12.443427 13875 mvcc.cc:205] Tried to move back new op lower bound from 7316481443570061312 to 7316481443141660672. Current Snapshot: MvccSnapshot[applied={T|T < 7316481443570061312}]
I20260809 04:39:12.453369 13135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:12.453598 13135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:12.456465 13135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b3cde0398ed40efb70b9a018e530fed: Sending DeleteTablet for 3 replicas of tablet b645f8b12b0c47228ad0b40f674bb35d
I20260809 04:39:12.457463 13326 tablet_service.cc:1514] Processing DeleteTablet for tablet b645f8b12b0c47228ad0b40f674bb35d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:36398
I20260809 04:39:12.457535 13728 tablet_service.cc:1514] Processing DeleteTablet for tablet b645f8b12b0c47228ad0b40f674bb35d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:51576
I20260809 04:39:12.458159 13889 tablet_replica.cc:330] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315: stopping tablet replica
I20260809 04:39:12.458289 13519 tablet_service.cc:1514] Processing DeleteTablet for tablet b645f8b12b0c47228ad0b40f674bb35d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:12 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:40832
I20260809 04:39:12.458539 13889 raft_consensus.cc:2229] T b645f8b12b0c47228ad0b40f674bb35d P 87e477eb455c4f9fbcb28fd4f1832315 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:39:12.458771 13892 tablet_replica.cc:330] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: stopping tablet replica
I20260809 04:39:12.458817 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 13227
I20260809 04:39:12.465649 13892 raft_consensus.cc:2229] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:12.466089 13892 raft_consensus.cc:2258] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:12.467690 13892 ts_tablet_manager.cc:1905] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:12.470450 13892 ts_tablet_manager.cc:1918] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:12.470621 13892 log.cc:1196] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/b645f8b12b0c47228ad0b40f674bb35d
I20260809 04:39:12.470945 13892 ts_tablet_manager.cc:1939] T b645f8b12b0c47228ad0b40f674bb35d P 6a75a808c7d44033b3cd1c5fadefe988: Deleting consensus metadata
I20260809 04:39:12.472301 13115 catalog_manager.cc:4841] TS 6a75a808c7d44033b3cd1c5fadefe988 (127.19.92.2:45719): tablet b645f8b12b0c47228ad0b40f674bb35d (table test-table [id=5463484cdf2b49f683ebba0911e7f9a8]) successfully deleted
I20260809 04:39:12.487884 13890 tablet_replica.cc:330] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: stopping tablet replica
I20260809 04:39:12.488276 13890 raft_consensus.cc:2229] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:12.488639 13890 raft_consensus.cc:2258] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:12.490054 13890 ts_tablet_manager.cc:1905] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:12.492362 13890 ts_tablet_manager.cc:1918] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:12.492465 13890 log.cc:1196] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/b645f8b12b0c47228ad0b40f674bb35d
I20260809 04:39:12.492739 13890 ts_tablet_manager.cc:1939] T b645f8b12b0c47228ad0b40f674bb35d P b557682c925d4f8eb2378377a09c51f8: Deleting consensus metadata
I20260809 04:39:12.493530 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 13430
W20260809 04:39:12.493546 13119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:39235 (error 108)
I20260809 04:39:12.502367 13116 catalog_manager.cc:4841] TS b557682c925d4f8eb2378377a09c51f8 (127.19.92.3:43147): tablet b645f8b12b0c47228ad0b40f674bb35d (table test-table [id=5463484cdf2b49f683ebba0911e7f9a8]) successfully deleted
I20260809 04:39:12.503448 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.503038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:39235 (local address 127.0.0.1:36408)
0809 04:39:12.503268 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:12.503345 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:39235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
W20260809 04:39:12.503633 13119 catalog_manager.cc:4584] TS 87e477eb455c4f9fbcb28fd4f1832315 (127.19.92.1:39235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b645f8b12b0c47228ad0b40f674bb35d: Network error: Client connection negotiation failed: client connection to 127.19.92.1:39235: connect: Connection refused (error 111)
I20260809 04:39:12.507525 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 13638
I20260809 04:39:12.519131 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 13083
2026-08-09T04:39:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2927 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T04:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:12Z Disabled control of system clock
I20260809 04:39:12.554903 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41067
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:39:12.668324 13902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:12.668603 13902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:12.668653 13902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:12.672608 13902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:12.672673 13902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:12.672694 13902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:12.672714 13902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:12.672734 13902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:12.677441 13902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:12.678557 13902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:12.680001 13902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:12.688745 13909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:12.688868 13902 server_base.cc:1034] running on GCE node
W20260809 04:39:12.688737 13913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:12.688740 13910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:12.689635 13902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:12.690788 13902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:12.691972 13902 hybrid_clock.cc:648] HybridClock initialized: now 1786250352691950 us; error 34 us; skew 500 ppm
I20260809 04:39:12.693889 13902 webserver.cc:460] Webserver started at http://127.19.92.62:42609/ using document root <none> and password file <none>
I20260809 04:39:12.694428 13902 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:12.694495 13902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:12.694675 13902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:12.696683 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "db0bff7f1ee341f48a594d46f11f7bc8"
format_stamp: "Formatted at 2026-08-09 04:39:12 on dist-test-slave-rmhg"
server_key: "0585e440b89b4634a6ce56c2f2b26b5e"
server_key_iv: "0d446aa83db6ae531d2265b44dd8a4c5"
server_key_version: "encryptionkey@0"
I20260809 04:39:12.697244 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "db0bff7f1ee341f48a594d46f11f7bc8"
format_stamp: "Formatted at 2026-08-09 04:39:12 on dist-test-slave-rmhg"
server_key: "0585e440b89b4634a6ce56c2f2b26b5e"
server_key_iv: "0d446aa83db6ae531d2265b44dd8a4c5"
server_key_version: "encryptionkey@0"
I20260809 04:39:12.700842 13902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:39:12.703315 13922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:12.704586 13902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:12.704702 13902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "db0bff7f1ee341f48a594d46f11f7bc8"
format_stamp: "Formatted at 2026-08-09 04:39:12 on dist-test-slave-rmhg"
server_key: "0585e440b89b4634a6ce56c2f2b26b5e"
server_key_iv: "0d446aa83db6ae531d2265b44dd8a4c5"
server_key_version: "encryptionkey@0"
I20260809 04:39:12.704797 13902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:12.731400 13902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:12.732149 13902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:12.732331 13902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:12.739253 13902 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41067
I20260809 04:39:12.739274 14007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41067 every 8 connection(s)
I20260809 04:39:12.740195 13902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:12.741816 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 13902
I20260809 04:39:12.741962 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:12.742302 19824 external_mini_cluster.cc:1428] Setting key 2fafce6a92b16c1e8ce47ce8d8984174
I20260809 04:39:12.744665 14008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:12.750684 14008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8: Bootstrap starting.
I20260809 04:39:12.751435 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.743621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.19.92.62:41067)
0809 04:39:12.744216 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:12.744231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:12.744266 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:39:12.744974 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:12.744996 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:12.745418 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:12.745731 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:12.746150 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.747133 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.748333 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.748814 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.748927 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:12.749752 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:12.749756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:12.749768 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:12.750135 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:12.750200 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:12.750220 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:12.750291 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:12.750439 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 04:39:12.750712 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260809 04:39:12.753078 14008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:12.753873 14008 log.cc:824] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:12.755779 14008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8: No bootstrap required, opened a new log
I20260809 04:39:12.759342 14008 raft_consensus.cc:348] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0bff7f1ee341f48a594d46f11f7bc8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41067 } }
I20260809 04:39:12.759567 14008 raft_consensus.cc:374] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:12.759608 14008 raft_consensus.cc:729] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0bff7f1ee341f48a594d46f11f7bc8, State: Initialized, Role: FOLLOWER
I20260809 04:39:12.760007 14008 consensus_queue.cc:260] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0bff7f1ee341f48a594d46f11f7bc8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41067 } }
I20260809 04:39:12.760129 14008 raft_consensus.cc:388] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:12.760174 14008 raft_consensus.cc:482] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:12.760244 14008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:12.761261 14008 raft_consensus.cc:504] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0bff7f1ee341f48a594d46f11f7bc8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41067 } }
I20260809 04:39:12.761656 14008 leader_election.cc:302] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db0bff7f1ee341f48a594d46f11f7bc8; no voters:
I20260809 04:39:12.761997 14008 leader_election.cc:288] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:12.762118 14015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:12.762336 14015 raft_consensus.cc:686] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [term 1 LEADER]: Becoming Leader. State: Replica: db0bff7f1ee341f48a594d46f11f7bc8, State: Running, Role: LEADER
I20260809 04:39:12.762686 14015 consensus_queue.cc:237] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0bff7f1ee341f48a594d46f11f7bc8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41067 } }
I20260809 04:39:12.763181 14008 sys_catalog.cc:564] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:12.764256 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db0bff7f1ee341f48a594d46f11f7bc8. Latest consensus state: current_term: 1 leader_uuid: "db0bff7f1ee341f48a594d46f11f7bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0bff7f1ee341f48a594d46f11f7bc8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41067 } } }
I20260809 04:39:12.764385 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:12.764791 14028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:12.764739 14017 sys_catalog.cc:455] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db0bff7f1ee341f48a594d46f11f7bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0bff7f1ee341f48a594d46f11f7bc8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41067 } } }
I20260809 04:39:12.765583 14017 sys_catalog.cc:458] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:12.768065 14028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:12.773666 14028 catalog_manager.cc:1347] Generated new cluster ID: 74c87ca86240424c90d19f0bb3364c4c
I20260809 04:39:12.773729 14028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:12.785981 14028 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:12.787312 14028 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:12.793632 14028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8: Generated new TSK 0
I20260809 04:39:12.794404 14028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:12.807853 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41067
--builtin_ntp_servers=127.19.92.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:39:12.921100 14044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:12.921425 14044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:12.921468 14044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:12.925625 14044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:12.925688 14044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:12.925743 14044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:39:12.930691 14044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:12.931834 14044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:12.933274 14044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:12.942785 14052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:12.942816 14053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:12.942911 14044 server_base.cc:1034] running on GCE node
W20260809 04:39:12.942803 14056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:12.943442 14044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:12.944089 14044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:12.945226 14044 hybrid_clock.cc:648] HybridClock initialized: now 1786250352945203 us; error 34 us; skew 500 ppm
I20260809 04:39:12.947006 14044 webserver.cc:460] Webserver started at http://127.19.92.1:38787/ using document root <none> and password file <none>
I20260809 04:39:12.947579 14044 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:12.947652 14044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:12.947860 14044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:12.949942 14044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "da02f41baabe4f78ae7a639d6d560acf"
format_stamp: "Formatted at 2026-08-09 04:39:12 on dist-test-slave-rmhg"
server_key: "0b9acc9f066366bbb9af2cab935a1452"
server_key_iv: "8b9a93386742d6ccd5cefebc83d5e278"
server_key_version: "encryptionkey@0"
I20260809 04:39:12.950479 14044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "da02f41baabe4f78ae7a639d6d560acf"
format_stamp: "Formatted at 2026-08-09 04:39:12 on dist-test-slave-rmhg"
server_key: "0b9acc9f066366bbb9af2cab935a1452"
server_key_iv: "8b9a93386742d6ccd5cefebc83d5e278"
server_key_version: "encryptionkey@0"
I20260809 04:39:12.954107 14044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:39:12.956612 14067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:12.957927 14044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:12.958047 14044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "da02f41baabe4f78ae7a639d6d560acf"
format_stamp: "Formatted at 2026-08-09 04:39:12 on dist-test-slave-rmhg"
server_key: "0b9acc9f066366bbb9af2cab935a1452"
server_key_iv: "8b9a93386742d6ccd5cefebc83d5e278"
server_key_version: "encryptionkey@0"
I20260809 04:39:12.958148 14044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:12.970840 14044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:12.971549 14044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:12.971735 14044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:12.972352 14044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:12.973353 14044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:12.973403 14044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:12.973456 14044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:12.973480 14044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:12.981782 14044 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:39821
I20260809 04:39:12.981808 14243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:39821 every 8 connection(s)
I20260809 04:39:12.982668 14044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:39:12.986004 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 14044
I20260809 04:39:12.986109 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:39:12.986373 19824 external_mini_cluster.cc:1428] Setting key 21b0e6b52c494c9193850681b9703e78
I20260809 04:39:12.988286 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41067
--builtin_ntp_servers=127.19.92.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:39:12.991703 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.984498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:40597 (local address 127.19.92.62:41067)
0809 04:39:12.984615 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:12.984619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:12.985676 (+ 1057us) server_negotiation.cc:407] Connection header received
0809 04:39:12.987098 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:12.987101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:12.987163 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:12.987250 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:12.988175 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.988749 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.989889 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.990120 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.990211 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:12.991227 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:12.991230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:12.991232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:12.991244 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:12.991264 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:12.991272 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:12.991329 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:12.991474 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 04:39:12.991560 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 04:39:12.992511 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:12.984891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41067 (local address 127.19.92.1:40597)
0809 04:39:12.985536 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:12.985575 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:12.986868 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:12.987391 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:12.987398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:12.987481 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:12.988039 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:12.988053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.988867 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.988870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:12.989768 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:12.989778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:12.990258 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:12.990262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:12.990417 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:12.990974 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:12.990975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:12.991018 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:12.991081 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:12.991339 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:12.991348 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:12.991354 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:39:12.991462 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 04:39:12.991750 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260809 04:39:12.993755 14246 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41067
I20260809 04:39:12.993974 14246 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:12.994470 14246 heartbeater.cc:511] Master 127.19.92.62:41067 requested a full tablet report, sending...
I20260809 04:39:12.995913 13951 ts_manager.cc:194] Registered new tserver with Master: da02f41baabe4f78ae7a639d6d560acf (127.19.92.1:39821)
I20260809 04:39:12.997077 13951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.1:40597
W20260809 04:39:13.100402 14250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:13.100692 14250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:13.100744 14250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:13.104666 14250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:13.104717 14250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:13.104785 14250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:39:13.110431 14250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:13.111512 14250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:13.112926 14250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:13.122483 14261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:13.122687 14250 server_base.cc:1034] running on GCE node
W20260809 04:39:13.122491 14258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:13.122490 14259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:13.123245 14250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:13.123941 14250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:13.125109 14250 hybrid_clock.cc:648] HybridClock initialized: now 1786250353125094 us; error 39 us; skew 500 ppm
I20260809 04:39:13.127034 14250 webserver.cc:460] Webserver started at http://127.19.92.2:37061/ using document root <none> and password file <none>
I20260809 04:39:13.127610 14250 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:13.127674 14250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:13.127880 14250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:13.130000 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "22a163bde46443358d3dabf34d2b0d9e"
format_stamp: "Formatted at 2026-08-09 04:39:13 on dist-test-slave-rmhg"
server_key: "57886674b2d65c1d747763186b6ba38c"
server_key_iv: "9aa562dd77e8ecf47d9a02a0fbd5e4af"
server_key_version: "encryptionkey@0"
I20260809 04:39:13.130548 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "22a163bde46443358d3dabf34d2b0d9e"
format_stamp: "Formatted at 2026-08-09 04:39:13 on dist-test-slave-rmhg"
server_key: "57886674b2d65c1d747763186b6ba38c"
server_key_iv: "9aa562dd77e8ecf47d9a02a0fbd5e4af"
server_key_version: "encryptionkey@0"
I20260809 04:39:13.134207 14250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:39:13.136647 14272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:13.137897 14250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:13.138011 14250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "22a163bde46443358d3dabf34d2b0d9e"
format_stamp: "Formatted at 2026-08-09 04:39:13 on dist-test-slave-rmhg"
server_key: "57886674b2d65c1d747763186b6ba38c"
server_key_iv: "9aa562dd77e8ecf47d9a02a0fbd5e4af"
server_key_version: "encryptionkey@0"
I20260809 04:39:13.138109 14250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:13.156816 14250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:13.157599 14250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:13.157781 14250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:13.158392 14250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:13.159395 14250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:13.159456 14250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:13.159511 14250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:13.159536 14250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:13.168269 14250 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:33043
I20260809 04:39:13.168319 14438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:33043 every 8 connection(s)
I20260809 04:39:13.169255 14250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:39:13.175766 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 14250
I20260809 04:39:13.175900 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:39:13.176172 19824 external_mini_cluster.cc:1428] Setting key 7da24c5e98fc76375e5d4932414189a6
I20260809 04:39:13.178196 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41067
--builtin_ntp_servers=127.19.92.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:39:13.178931 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.171148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:41555 (local address 127.19.92.62:41067)
0809 04:39:13.171297 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.171301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.172398 (+ 1097us) server_negotiation.cc:407] Connection header received
0809 04:39:13.173876 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.173878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.173929 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.174009 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:13.174976 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.175572 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.176880 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.177217 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.177332 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.178465 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:13.178469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:13.178471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:13.178488 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:13.178511 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:13.178520 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:13.178592 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.178712 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.178781 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:39:13.179802 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.171563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41067 (local address 127.19.92.2:41555)
0809 04:39:13.172239 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:13.172281 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:13.173673 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:13.174164 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:13.174171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:13.174258 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:13.174801 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.174814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.175713 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.175717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.176745 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.176756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.177360 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.177363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.177593 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.178163 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:13.178164 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:13.178206 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:13.178273 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:13.178594 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:13.178599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:13.178604 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 04:39:13.178699 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 04:39:13.178931 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260809 04:39:13.180958 14439 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41067
I20260809 04:39:13.181212 14439 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:13.181710 14439 heartbeater.cc:511] Master 127.19.92.62:41067 requested a full tablet report, sending...
I20260809 04:39:13.182706 13951 ts_manager.cc:194] Registered new tserver with Master: 22a163bde46443358d3dabf34d2b0d9e (127.19.92.2:33043)
I20260809 04:39:13.183264 13951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.2:41555
W20260809 04:39:13.295192 14445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:13.295468 14445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:13.295511 14445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:13.299613 14445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:13.299678 14445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:13.299734 14445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:39:13.304649 14445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:13.305855 14445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:13.307278 14445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:13.317180 14458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:13.317186 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:13.317262 14445 server_base.cc:1034] running on GCE node
W20260809 04:39:13.317359 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:13.317888 14445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:13.318609 14445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:13.319798 14445 hybrid_clock.cc:648] HybridClock initialized: now 1786250353319782 us; error 32 us; skew 500 ppm
I20260809 04:39:13.321962 14445 webserver.cc:460] Webserver started at http://127.19.92.3:46369/ using document root <none> and password file <none>
I20260809 04:39:13.322609 14445 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:13.322687 14445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:13.322908 14445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:13.325285 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "ace7647e795f4d3dbf92cb3373016acf"
format_stamp: "Formatted at 2026-08-09 04:39:13 on dist-test-slave-rmhg"
server_key: "31e42357c82cec676a3976f3efef9b81"
server_key_iv: "39f3b5969e446f51928e7bcadbac3a5f"
server_key_version: "encryptionkey@0"
I20260809 04:39:13.325906 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "ace7647e795f4d3dbf92cb3373016acf"
format_stamp: "Formatted at 2026-08-09 04:39:13 on dist-test-slave-rmhg"
server_key: "31e42357c82cec676a3976f3efef9b81"
server_key_iv: "39f3b5969e446f51928e7bcadbac3a5f"
server_key_version: "encryptionkey@0"
I20260809 04:39:13.329919 14445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:39:13.332616 14468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:13.334003 14445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:13.334131 14445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "ace7647e795f4d3dbf92cb3373016acf"
format_stamp: "Formatted at 2026-08-09 04:39:13 on dist-test-slave-rmhg"
server_key: "31e42357c82cec676a3976f3efef9b81"
server_key_iv: "39f3b5969e446f51928e7bcadbac3a5f"
server_key_version: "encryptionkey@0"
I20260809 04:39:13.334246 14445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:13.358952 14445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:13.359745 14445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:13.359938 14445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:13.360603 14445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:13.361764 14445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:13.361817 14445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:13.361883 14445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:13.361907 14445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:13.371017 14445 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:33579
I20260809 04:39:13.371052 14629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:33579 every 8 connection(s)
I20260809 04:39:13.371977 14445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:39:13.376154 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 14445
I20260809 04:39:13.376266 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:39:13.376551 19824 external_mini_cluster.cc:1428] Setting key 1bce097de206c64d40135cd9c5c5b1ab
I20260809 04:39:13.381810 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.373869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:38011 (local address 127.19.92.62:41067)
0809 04:39:13.374043 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.374047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.375160 (+ 1113us) server_negotiation.cc:407] Connection header received
0809 04:39:13.376803 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.376807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.376869 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.376952 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:13.378210 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.378847 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.380009 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.380219 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.380289 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.381429 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:13.381432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:13.381434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:13.381447 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:13.381465 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:13.381473 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:13.381532 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.381647 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.381696 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 04:39:13.382748 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.374310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41067 (local address 127.19.92.3:38011)
0809 04:39:13.375008 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:13.375051 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:13.376551 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:13.377214 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:13.377225 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:13.377344 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:13.378059 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.378082 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.378981 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.378984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.379908 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.379918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.380398 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.380401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.380569 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.381172 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:13.381174 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:13.381218 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:13.381289 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:13.381544 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:13.381549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:13.381553 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:39:13.381644 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 04:39:13.381878 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260809 04:39:13.383790 14631 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41067
I20260809 04:39:13.384028 14631 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:13.384588 14631 heartbeater.cc:511] Master 127.19.92.62:41067 requested a full tablet report, sending...
I20260809 04:39:13.385607 13951 ts_manager.cc:194] Registered new tserver with Master: ace7647e795f4d3dbf92cb3373016acf (127.19.92.3:33579)
I20260809 04:39:13.386168 13951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.3:38011
I20260809 04:39:13.390368 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:39:13.395895 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.392314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.19.92.62:41067)
0809 04:39:13.392440 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.392443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.392573 (+ 130us) server_negotiation.cc:407] Connection header received
0809 04:39:13.392672 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.392674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.392713 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.392782 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:13.393024 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.393617 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.394449 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.394711 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.394794 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.395546 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:13.395549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:13.395551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:13.395565 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:13.395586 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:13.395594 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:13.395654 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.395726 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.395778 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:39:13.398970 13951 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:39:13.402164 13951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:39:13.417799 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.411815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.19.92.3:33579)
0809 04:39:13.412955 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.412970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.412995 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:39:13.413077 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.413082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.413326 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.413483 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:13.413783 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.414952 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.416463 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.417052 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.417177 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.417389 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.417502 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.417605 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":124,"threads_started":1}
I20260809 04:39:13.417805 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.411687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:33579 (local address 127.0.0.1:37606)
0809 04:39:13.412453 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:13.412511 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:13.412648 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:13.413482 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:13.413488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:13.413511 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:13.413653 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.413659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.415065 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.415068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.416321 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.416332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.417192 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.417196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.417305 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.417345 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 04:39:13.417470 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 04:39:13.417602 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":55,"spinlock_wait_cycles":11520,"thread_start_us":95,"threads_started":1}
I20260809 04:39:13.417997 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.412512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33043 (local address 127.0.0.1:36806)
0809 04:39:13.412865 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:13.412876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:13.412965 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:13.413907 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:13.413910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:13.413939 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:13.414037 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.414044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.415373 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.415377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.416388 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.416396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.417649 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.417653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.417743 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.417766 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:39:13.417829 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 04:39:13.417894 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":106,"threads_started":1}
I20260809 04:39:13.417831 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.411729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:39821 (local address 127.0.0.1:46822)
0809 04:39:13.412453 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:13.412511 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:13.412648 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:13.413723 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:13.413726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:13.413746 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:13.413845 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.413851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.414873 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.414876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.416279 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.416290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.417415 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.417418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.417507 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.417531 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:39:13.417582 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 04:39:13.417653 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":38,"spinlock_wait_cycles":1280,"thread_start_us":126,"threads_started":1}
I20260809 04:39:13.418423 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.412021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.19.92.1:39821)
0809 04:39:13.413290 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.413302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.413326 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:39:13.413401 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.413406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.413589 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.413741 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:13.413942 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.414754 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.416417 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.417274 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.417900 (+ 626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.418105 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.418161 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.418257 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":114,"threads_started":1}
I20260809 04:39:13.418430 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.412765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.19.92.2:33043)
0809 04:39:13.413289 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.413302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.413326 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:39:13.413401 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.413406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.413589 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.413741 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:13.414170 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.415247 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.416509 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.417281 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.417384 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.417591 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.418173 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.418261 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":171,"threads_started":1}
I20260809 04:39:13.421689 14152 tablet_service.cc:1467] Processing CreateTablet for tablet 4639aa6371e74ae7b5508c39432254e2 (DEFAULT_TABLE table=test-table [id=a14068b1e48b45dfbc673231dbafcc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:13.421550 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 4639aa6371e74ae7b5508c39432254e2 (DEFAULT_TABLE table=test-table [id=a14068b1e48b45dfbc673231dbafcc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:13.421550 14344 tablet_service.cc:1467] Processing CreateTablet for tablet 4639aa6371e74ae7b5508c39432254e2 (DEFAULT_TABLE table=test-table [id=a14068b1e48b45dfbc673231dbafcc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:13.422653 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4639aa6371e74ae7b5508c39432254e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:13.422674 14344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4639aa6371e74ae7b5508c39432254e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:13.422674 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4639aa6371e74ae7b5508c39432254e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:13.429092 14661 tablet_bootstrap.cc:492] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf: Bootstrap starting.
I20260809 04:39:13.430789 14662 tablet_bootstrap.cc:492] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Bootstrap starting.
I20260809 04:39:13.431447 14661 tablet_bootstrap.cc:654] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:13.432190 14661 log.cc:824] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:13.432667 14662 tablet_bootstrap.cc:654] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:13.432868 14663 tablet_bootstrap.cc:492] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Bootstrap starting.
I20260809 04:39:13.433485 14662 log.cc:824] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:13.434818 14663 tablet_bootstrap.cc:654] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:13.435312 14661 tablet_bootstrap.cc:492] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf: No bootstrap required, opened a new log
I20260809 04:39:13.435479 14663 log.cc:824] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:13.435520 14661 ts_tablet_manager.cc:1397] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 04:39:13.435590 14662 tablet_bootstrap.cc:492] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: No bootstrap required, opened a new log
I20260809 04:39:13.435820 14662 ts_tablet_manager.cc:1397] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:39:13.437124 14663 tablet_bootstrap.cc:492] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: No bootstrap required, opened a new log
I20260809 04:39:13.437350 14663 ts_tablet_manager.cc:1397] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 04:39:13.440042 14661 raft_consensus.cc:348] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.440385 14661 raft_consensus.cc:374] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:13.440466 14661 raft_consensus.cc:729] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da02f41baabe4f78ae7a639d6d560acf, State: Initialized, Role: FOLLOWER
I20260809 04:39:13.440433 14662 raft_consensus.cc:348] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.440707 14662 raft_consensus.cc:374] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:13.440793 14662 raft_consensus.cc:729] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a163bde46443358d3dabf34d2b0d9e, State: Initialized, Role: FOLLOWER
I20260809 04:39:13.440639 14663 raft_consensus.cc:348] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.440924 14663 raft_consensus.cc:374] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:13.440999 14663 raft_consensus.cc:729] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace7647e795f4d3dbf92cb3373016acf, State: Initialized, Role: FOLLOWER
I20260809 04:39:13.441047 14661 consensus_queue.cc:260] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.441222 14662 consensus_queue.cc:260] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.441526 14663 consensus_queue.cc:260] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.442296 14631 heartbeater.cc:503] Master 127.19.92.62:41067 was elected leader, sending a full tablet report...
I20260809 04:39:13.442322 14662 ts_tablet_manager.cc:1428] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:39:13.442322 14661 ts_tablet_manager.cc:1428] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 04:39:13.442679 14663 ts_tablet_manager.cc:1428] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:39:13.442688 14439 heartbeater.cc:503] Master 127.19.92.62:41067 was elected leader, sending a full tablet report...
I20260809 04:39:13.443863 14246 heartbeater.cc:503] Master 127.19.92.62:41067 was elected leader, sending a full tablet report...
W20260809 04:39:13.483841 14247 tablet.cc:2369] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:39:13.623025 14633 tablet.cc:2369] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:39:13.670413 14440 tablet.cc:2369] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:13.675308 14669 raft_consensus.cc:482] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:13.675514 14669 raft_consensus.cc:504] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.676721 14669 leader_election.cc:288] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da02f41baabe4f78ae7a639d6d560acf (127.19.92.1:39821), 22a163bde46443358d3dabf34d2b0d9e (127.19.92.2:33043)
I20260809 04:39:13.680537 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.677168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:40523 (local address 127.19.92.2:33043)
0809 04:39:13.677301 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.677307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.677326 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:39:13.677506 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.677509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.677571 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.677649 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:13.677939 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.678518 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.679546 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.679950 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.680030 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.680093 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.680317 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.680393 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 04:39:13.680554 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.677033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:39821 (local address 127.19.92.3:42083)
0809 04:39:13.677493 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:13.677507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:13.677607 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:13.677848 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:13.677850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:13.677864 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:13.678000 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.678006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.678697 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.678699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.679486 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.679497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.680137 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.680140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.680239 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.680258 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:39:13.680330 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 04:39:13.680392 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":20,"thread_start_us":145,"threads_started":1}
I20260809 04:39:13.680562 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.677112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:42083 (local address 127.19.92.1:39821)
0809 04:39:13.677323 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.677327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.677591 (+ 264us) server_negotiation.cc:407] Connection header received
0809 04:39:13.677704 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.677707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.677755 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.677830 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:13.678108 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.678591 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.679617 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.680007 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.680108 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.680169 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.680364 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.680440 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260809 04:39:13.680573 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.677013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:33043 (local address 127.19.92.3:40523)
0809 04:39:13.677239 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:13.677255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:13.677360 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:13.677670 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:13.677673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:13.677693 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:13.677830 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.677838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.678626 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.678629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.679425 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:13.679435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.680071 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.680075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:13.680179 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.680226 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 04:39:13.680283 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 04:39:13.680359 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":3}
I20260809 04:39:13.681473 14179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4639aa6371e74ae7b5508c39432254e2" candidate_uuid: "ace7647e795f4d3dbf92cb3373016acf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da02f41baabe4f78ae7a639d6d560acf" is_pre_election: true
I20260809 04:39:13.681694 14369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4639aa6371e74ae7b5508c39432254e2" candidate_uuid: "ace7647e795f4d3dbf92cb3373016acf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a163bde46443358d3dabf34d2b0d9e" is_pre_election: true
I20260809 04:39:13.681862 14179 raft_consensus.cc:2454] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace7647e795f4d3dbf92cb3373016acf in term 0.
I20260809 04:39:13.682020 14369 raft_consensus.cc:2454] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace7647e795f4d3dbf92cb3373016acf in term 0.
I20260809 04:39:13.682382 14472 leader_election.cc:302] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ace7647e795f4d3dbf92cb3373016acf, da02f41baabe4f78ae7a639d6d560acf; no voters:
I20260809 04:39:13.682651 14669 raft_consensus.cc:2781] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:39:13.682740 14669 raft_consensus.cc:482] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:39:13.682796 14669 raft_consensus.cc:3037] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:13.683840 14669 raft_consensus.cc:504] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.684207 14669 leader_election.cc:288] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [CANDIDATE]: Term 1 election: Requested vote from peers da02f41baabe4f78ae7a639d6d560acf (127.19.92.1:39821), 22a163bde46443358d3dabf34d2b0d9e (127.19.92.2:33043)
I20260809 04:39:13.684592 14369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4639aa6371e74ae7b5508c39432254e2" candidate_uuid: "ace7647e795f4d3dbf92cb3373016acf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a163bde46443358d3dabf34d2b0d9e"
I20260809 04:39:13.684613 14179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4639aa6371e74ae7b5508c39432254e2" candidate_uuid: "ace7647e795f4d3dbf92cb3373016acf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da02f41baabe4f78ae7a639d6d560acf"
I20260809 04:39:13.684712 14369 raft_consensus.cc:3037] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:13.684724 14179 raft_consensus.cc:3037] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:13.685798 14179 raft_consensus.cc:2454] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace7647e795f4d3dbf92cb3373016acf in term 1.
I20260809 04:39:13.685817 14369 raft_consensus.cc:2454] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace7647e795f4d3dbf92cb3373016acf in term 1.
I20260809 04:39:13.686120 14477 leader_election.cc:302] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22a163bde46443358d3dabf34d2b0d9e, ace7647e795f4d3dbf92cb3373016acf, da02f41baabe4f78ae7a639d6d560acf; no voters:
I20260809 04:39:13.686386 14669 raft_consensus.cc:2781] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:13.686599 14669 raft_consensus.cc:686] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 1 LEADER]: Becoming Leader. State: Replica: ace7647e795f4d3dbf92cb3373016acf, State: Running, Role: LEADER
I20260809 04:39:13.686964 14669 consensus_queue.cc:237] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } }
I20260809 04:39:13.690192 13951 catalog_manager.cc:5495] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf reported cstate change: term changed from 0 to 1, leader changed from <none> to ace7647e795f4d3dbf92cb3373016acf (127.19.92.3). New cstate: current_term: 1 leader_uuid: "ace7647e795f4d3dbf92cb3373016acf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ace7647e795f4d3dbf92cb3373016acf" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 33579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260809 04:39:13.710597 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.707179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.19.92.3:33579)
0809 04:39:13.707300 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:13.707305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:13.707338 (+ 33us) server_negotiation.cc:407] Connection header received
0809 04:39:13.707410 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:13.707412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:13.707462 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:13.707550 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:13.707793 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.708336 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.709192 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:13.709409 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:13.709497 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:13.709803 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:13.710217 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:13.710321 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:13.710394 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 04:39:13.710446 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 04:39:13.716329 14179 raft_consensus.cc:1261] T 4639aa6371e74ae7b5508c39432254e2 P da02f41baabe4f78ae7a639d6d560acf [term 1 FOLLOWER]: Refusing update from remote peer ace7647e795f4d3dbf92cb3373016acf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:13.716329 14369 raft_consensus.cc:1261] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 1 FOLLOWER]: Refusing update from remote peer ace7647e795f4d3dbf92cb3373016acf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:13.717026 14677 consensus_queue.cc:1035] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [LEADER]: Connected to new peer: Peer: permanent_uuid: "da02f41baabe4f78ae7a639d6d560acf" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:13.717276 14669 consensus_queue.cc:1035] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a163bde46443358d3dabf34d2b0d9e" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:13.726171 14688 mvcc.cc:205] Tried to move back new op lower bound from 7316481448815132672 to 7316481448704557056. Current Snapshot: MvccSnapshot[applied={T|T < 7316481448815132672}]
I20260809 04:39:13.734607 13951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:13.734797 13951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:13.737270 13951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db0bff7f1ee341f48a594d46f11f7bc8: Sending DeleteTablet for 3 replicas of tablet 4639aa6371e74ae7b5508c39432254e2
I20260809 04:39:13.738145 14152 tablet_service.cc:1514] Processing DeleteTablet for tablet 4639aa6371e74ae7b5508c39432254e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:13 UTC) from {username='slave'} at 127.0.0.1:46822
I20260809 04:39:13.738356 14344 tablet_service.cc:1514] Processing DeleteTablet for tablet 4639aa6371e74ae7b5508c39432254e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:13 UTC) from {username='slave'} at 127.0.0.1:36806
I20260809 04:39:13.738857 14701 tablet_replica.cc:330] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: stopping tablet replica
I20260809 04:39:13.739073 14539 tablet_service.cc:1514] Processing DeleteTablet for tablet 4639aa6371e74ae7b5508c39432254e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:13 UTC) from {username='slave'} at 127.0.0.1:37606
I20260809 04:39:13.739363 14701 raft_consensus.cc:2229] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:13.739563 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 14044
I20260809 04:39:13.739919 14701 raft_consensus.cc:2258] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:13.747936 14701 ts_tablet_manager.cc:1905] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:13.751637 14701 ts_tablet_manager.cc:1918] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:13.751876 14701 log.cc:1196] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/4639aa6371e74ae7b5508c39432254e2
I20260809 04:39:13.752295 14701 ts_tablet_manager.cc:1939] T 4639aa6371e74ae7b5508c39432254e2 P 22a163bde46443358d3dabf34d2b0d9e: Deleting consensus metadata
I20260809 04:39:13.753995 13933 catalog_manager.cc:4841] TS 22a163bde46443358d3dabf34d2b0d9e (127.19.92.2:33043): tablet 4639aa6371e74ae7b5508c39432254e2 (table test-table [id=a14068b1e48b45dfbc673231dbafcc7e]) successfully deleted
I20260809 04:39:13.761471 14702 tablet_replica.cc:330] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: stopping tablet replica
I20260809 04:39:13.761912 14702 raft_consensus.cc:2229] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:39:13.762522 14702 raft_consensus.cc:2258] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:13.764597 14702 ts_tablet_manager.cc:1905] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:13.767398 14702 ts_tablet_manager.cc:1918] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:13.767513 14702 log.cc:1196] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/4639aa6371e74ae7b5508c39432254e2
I20260809 04:39:13.767791 14702 ts_tablet_manager.cc:1939] T 4639aa6371e74ae7b5508c39432254e2 P ace7647e795f4d3dbf92cb3373016acf: Deleting consensus metadata
W20260809 04:39:13.768421 13928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:39821 (error 108)
I20260809 04:39:13.769034 13933 catalog_manager.cc:4841] TS ace7647e795f4d3dbf92cb3373016acf (127.19.92.3:33579): tablet 4639aa6371e74ae7b5508c39432254e2 (table test-table [id=a14068b1e48b45dfbc673231dbafcc7e]) successfully deleted
I20260809 04:39:13.769671 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:13.769338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:39821 (local address 127.0.0.1:46830)
0809 04:39:13.769508 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:13.769578 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 04:39:13.769784 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 14250
W20260809 04:39:13.769914 13928 catalog_manager.cc:4584] TS da02f41baabe4f78ae7a639d6d560acf (127.19.92.1:39821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4639aa6371e74ae7b5508c39432254e2: Network error: Client connection negotiation failed: client connection to 127.19.92.1:39821: connect: Connection refused (error 111)
I20260809 04:39:13.795023 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 14445
I20260809 04:39:13.808220 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 13902
2026-08-09T04:39:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1291 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:13 dist-test-slave-rmhg krb5kdc[14709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:13 dist-test-slave-rmhg krb5kdc[14709](info): set up 2 sockets
Aug 09 04:39:13 dist-test-slave-rmhg krb5kdc[14709](info): commencing operation
krb5kdc: starting...
W20260809 04:39:14.936175 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:15.063791 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:34545
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:34545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:15.181052 14734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:15.181406 14734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:15.181454 14734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:15.185534 14734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:15.185612 14734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:15.185632 14734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:15.185648 14734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:15.185662 14734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:15.191190 14734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:34545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:34545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:15.192335 14734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:15.193439 14734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:39:15.202292 14734 server_base.cc:1034] running on GCE node
W20260809 04:39:15.202248 14744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:15.202248 14747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:15.202219 14741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:15.203135 14734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:15.204351 14734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:15.205556 14734 hybrid_clock.cc:648] HybridClock initialized: now 1786250355205527 us; error 45 us; skew 500 ppm
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:15.208529 14734 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:15.209735 14734 webserver.cc:460] Webserver started at http://127.19.92.62:43693/ using document root <none> and password file <none>
I20260809 04:39:15.210376 14734 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:15.210487 14734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:15.210724 14734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:15.212888 14734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "b766b0c35bf54fe8a0f9547600bd57c2"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "471f72b539f27b03357ab59489256f60"
server_key_iv: "4e4f7b94656f7d0d1eaf86e2399ba575"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.213498 14734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "b766b0c35bf54fe8a0f9547600bd57c2"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "471f72b539f27b03357ab59489256f60"
server_key_iv: "4e4f7b94656f7d0d1eaf86e2399ba575"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.217453 14734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:39:15.220062 14756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.221556 14734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:15.221693 14734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "b766b0c35bf54fe8a0f9547600bd57c2"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "471f72b539f27b03357ab59489256f60"
server_key_iv: "4e4f7b94656f7d0d1eaf86e2399ba575"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.221807 14734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:15.232784 14734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:15.236307 14734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:15.236526 14734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:15.243950 14734 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:34545
I20260809 04:39:15.244103 14831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:34545 every 8 connection(s)
I20260809 04:39:15.245126 14734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:15.248337 14833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:15.251024 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 14734
I20260809 04:39:15.251148 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:15.251425 19824 external_mini_cluster.cc:1428] Setting key 6d35589f13d851291f509fbea30f454a
I20260809 04:39:15.254709 14833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2: Bootstrap starting.
I20260809 04:39:15.257323 14833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:15.258232 14833 log.cc:824] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:15.260478 14833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2: No bootstrap required, opened a new log
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:15.264647 14833 raft_consensus.cc:348] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b766b0c35bf54fe8a0f9547600bd57c2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34545 } }
I20260809 04:39:15.264941 14833 raft_consensus.cc:374] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:15.265005 14833 raft_consensus.cc:729] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b766b0c35bf54fe8a0f9547600bd57c2, State: Initialized, Role: FOLLOWER
I20260809 04:39:15.265550 14833 consensus_queue.cc:260] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b766b0c35bf54fe8a0f9547600bd57c2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34545 } }
I20260809 04:39:15.265723 14833 raft_consensus.cc:388] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:15.265842 14833 raft_consensus.cc:482] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:15.265949 14833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:15.267103 14833 raft_consensus.cc:504] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b766b0c35bf54fe8a0f9547600bd57c2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34545 } }
I20260809 04:39:15.267169 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:15.252901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.19.92.62:34545)
0809 04:39:15.253380 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:15.253398 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:15.253437 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:39:15.254102 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:15.254121 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:15.254490 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:15.254799 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:15.255730 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.256755 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.257981 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.258538 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.258666 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:15.261457 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:15.261464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:15.261483 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:15.261518 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:15.263938 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:15.264391 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:15.264397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:15.264402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:15.264565 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:15.264803 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:15.264806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:15.264808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:15.265378 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:15.265591 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:15.265817 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 04:39:15.266149 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":165,"threads_started":1}
I20260809 04:39:15.267504 14833 leader_election.cc:302] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b766b0c35bf54fe8a0f9547600bd57c2; no voters:
I20260809 04:39:15.267829 14833 leader_election.cc:288] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:15.267935 14841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:15.268157 14841 raft_consensus.cc:686] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [term 1 LEADER]: Becoming Leader. State: Replica: b766b0c35bf54fe8a0f9547600bd57c2, State: Running, Role: LEADER
I20260809 04:39:15.268533 14841 consensus_queue.cc:237] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b766b0c35bf54fe8a0f9547600bd57c2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34545 } }
I20260809 04:39:15.269050 14833 sys_catalog.cc:564] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:15.269896 14843 sys_catalog.cc:455] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b766b0c35bf54fe8a0f9547600bd57c2. Latest consensus state: current_term: 1 leader_uuid: "b766b0c35bf54fe8a0f9547600bd57c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b766b0c35bf54fe8a0f9547600bd57c2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34545 } } }
I20260809 04:39:15.270146 14843 sys_catalog.cc:458] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:15.270284 14842 sys_catalog.cc:455] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b766b0c35bf54fe8a0f9547600bd57c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b766b0c35bf54fe8a0f9547600bd57c2" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34545 } } }
I20260809 04:39:15.270449 14842 sys_catalog.cc:458] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2 [sys.catalog]: This master's current role is: LEADER
W20260809 04:39:15.273653 14867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:15.273725 14867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:15.273804 14856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:15.274497 14856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:15.280229 14856 catalog_manager.cc:1347] Generated new cluster ID: 2b2ef458c47b44f8b6d69f97b2e9a2c0
I20260809 04:39:15.280326 14856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:15.294502 14856 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:15.295483 14856 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:15.300060 14856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2: Generated new TSK 0
I20260809 04:39:15.300748 14856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:39:15.339802 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34545
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:15.456640 14873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:15.456935 14873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:15.456998 14873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:15.461058 14873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:15.461133 14873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:15.461239 14873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:39:15.466583 14873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:15.467700 14873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:15.468775 14873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:15.478468 14878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:15.478478 14883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:15.478717 14873 server_base.cc:1034] running on GCE node
W20260809 04:39:15.478483 14879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:15.479238 14873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:15.479923 14873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:15.481078 14873 hybrid_clock.cc:648] HybridClock initialized: now 1786250355481059 us; error 31 us; skew 500 ppm
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:15.484079 14873 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:39:15.485118 14873 webserver.cc:460] Webserver started at http://127.19.92.1:38087/ using document root <none> and password file <none>
I20260809 04:39:15.485776 14873 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:15.485845 14873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:15.486040 14873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:15.488245 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "ebbac262fbe247a38e9eaea6e1ccdaea"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "951f4c0c1a81b37a38aa8cfd7cfac8be"
server_key_iv: "847c1fd8f717d7034d17a48973ebf270"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.488834 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "ebbac262fbe247a38e9eaea6e1ccdaea"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "951f4c0c1a81b37a38aa8cfd7cfac8be"
server_key_iv: "847c1fd8f717d7034d17a48973ebf270"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.492720 14873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:39:15.495329 14891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.496541 14873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:15.496665 14873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "ebbac262fbe247a38e9eaea6e1ccdaea"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "951f4c0c1a81b37a38aa8cfd7cfac8be"
server_key_iv: "847c1fd8f717d7034d17a48973ebf270"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.496763 14873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:15.521308 14873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:15.524874 14873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:15.525102 14873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:15.525894 14873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:15.526876 14873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:15.526938 14873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.526981 14873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:15.527001 14873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.535645 14873 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:46291
I20260809 04:39:15.535702 15065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:46291 every 8 connection(s)
I20260809 04:39:15.536598 14873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:39:15.538100 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 14873
I20260809 04:39:15.538221 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:39:15.538556 19824 external_mini_cluster.cc:1428] Setting key bf35662630ab99501280a6d756d0e294
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:39:15.550544 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:15.538522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:41857 (local address 127.19.92.62:34545)
0809 04:39:15.538682 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:15.538686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:15.539608 (+ 922us) server_negotiation.cc:407] Connection header received
0809 04:39:15.541069 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:15.541073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:15.541127 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:15.541226 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:15.542461 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.543005 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.544160 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.544387 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.544492 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:15.547091 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:15.547094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:15.547097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:15.547130 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:15.548540 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:15.549009 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:15.549012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:15.549013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:15.549100 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:15.549461 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:15.549463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:15.549465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:15.549655 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:15.549729 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:15.550256 (+ 527us) server_negotiation.cc:299] Negotiation successful
0809 04:39:15.550386 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:39:15.551553 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:15.538909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34545 (local address 127.19.92.1:41857)
0809 04:39:15.539446 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:15.539507 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:15.540848 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:15.541412 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:15.541419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:15.541789 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:15.542305 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:15.542319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.543134 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.543137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:15.544037 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:15.544047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.544547 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.544551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:15.544727 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:15.545316 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:15.545333 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:15.546891 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:15.548648 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:15.548654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:15.548666 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:15.548917 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:15.549220 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:15.549222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:15.549224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:15.549361 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:15.549729 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:15.549734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:15.549979 (+ 245us) client_negotiation.cc:769] Sending connection context
0809 04:39:15.550195 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 04:39:15.550508 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":135,"threads_started":1}
I20260809 04:39:15.552907 15066 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34545
I20260809 04:39:15.553184 15066 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:15.553726 15066 heartbeater.cc:511] Master 127.19.92.62:34545 requested a full tablet report, sending...
I20260809 04:39:15.555413 14773 ts_manager.cc:194] Registered new tserver with Master: ebbac262fbe247a38e9eaea6e1ccdaea (127.19.92.1:46291)
I20260809 04:39:15.556597 14773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:41857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:39:15.566669 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34545
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:15.681546 15072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:15.681816 15072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:15.681859 15072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:15.686044 15072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:15.686131 15072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:15.686218 15072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:39:15.691766 15072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:15.692878 15072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:15.693948 15072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:15.703630 15077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:15.703630 15078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:15.703714 15081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:15.703949 15072 server_base.cc:1034] running on GCE node
I20260809 04:39:15.704348 15072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:15.704994 15072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:15.706142 15072 hybrid_clock.cc:648] HybridClock initialized: now 1786250355706125 us; error 63 us; skew 500 ppm
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:15.708740 15072 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:39:15.709661 15072 webserver.cc:460] Webserver started at http://127.19.92.2:36811/ using document root <none> and password file <none>
I20260809 04:39:15.710191 15072 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:15.710245 15072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:15.710417 15072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:15.712410 15072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "0977ce174f3b44dda924b29cf2daf681"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "8090ac7ee0bf733cf12c87bd9c8b3a3f"
server_key_iv: "04cd871363770b535c383425ec15052f"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.712913 15072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "0977ce174f3b44dda924b29cf2daf681"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "8090ac7ee0bf733cf12c87bd9c8b3a3f"
server_key_iv: "04cd871363770b535c383425ec15052f"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.716593 15072 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:39:15.719017 15090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.720286 15072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:39:15.720386 15072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "0977ce174f3b44dda924b29cf2daf681"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "8090ac7ee0bf733cf12c87bd9c8b3a3f"
server_key_iv: "04cd871363770b535c383425ec15052f"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.720474 15072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:15.737236 15072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:15.740425 15072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:15.740671 15072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:15.741298 15072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:15.742173 15072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:15.742221 15072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.742274 15072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:15.742303 15072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.750524 15072 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:34863
I20260809 04:39:15.750589 15258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:34863 every 8 connection(s)
I20260809 04:39:15.751421 15072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:39:15.754716 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 15072
I20260809 04:39:15.754873 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:39:15.755285 19824 external_mini_cluster.cc:1428] Setting key aaba8654ca955916db06ad97b6a11015
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:39:15.770133 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:15.753318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:51873 (local address 127.19.92.62:34545)
0809 04:39:15.753444 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:15.753448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:15.755994 (+ 2546us) server_negotiation.cc:407] Connection header received
0809 04:39:15.757839 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:15.757843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:15.757940 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:15.758115 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:15.759978 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.760749 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.762134 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.762502 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.762641 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:15.765880 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:15.765885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:15.765889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:15.765924 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:15.767607 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:15.768239 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:15.768242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:15.768244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:15.768338 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:15.768780 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:15.768784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:15.768785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:15.769051 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:15.769187 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:15.769863 (+ 676us) server_negotiation.cc:299] Negotiation successful
0809 04:39:15.769999 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:39:15.771122 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:15.753817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34545 (local address 127.19.92.2:51873)
0809 04:39:15.755810 (+ 1993us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:15.755878 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:15.757510 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:15.758321 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:15.758331 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:15.758852 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:15.759797 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:15.759819 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.760878 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.760881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:15.761994 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:15.762005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.762897 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.762900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:15.763081 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:15.763720 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:15.763739 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:15.765617 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:15.767748 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:15.767754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:15.767769 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:15.768106 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:15.768466 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:15.768469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:15.768471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:15.768658 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:15.769232 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:15.769239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:15.769560 (+ 321us) client_negotiation.cc:769] Sending connection context
0809 04:39:15.769749 (+ 189us) client_negotiation.cc:240] Negotiation successful
0809 04:39:15.770138 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:39:15.772444 15259 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34545
I20260809 04:39:15.772706 15259 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:15.773269 15259 heartbeater.cc:511] Master 127.19.92.62:34545 requested a full tablet report, sending...
I20260809 04:39:15.774425 14773 ts_manager.cc:194] Registered new tserver with Master: 0977ce174f3b44dda924b29cf2daf681 (127.19.92.2:34863)
I20260809 04:39:15.775125 14773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:51873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:39:15.789062 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34545
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:15.902477 15264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:15.902768 15264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:15.902832 15264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:15.906960 15264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:15.907027 15264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:15.907107 15264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:39:15.912480 15264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:15.913616 15264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:15.914642 15264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:15.924615 15273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:15.924615 15275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:15.924618 15272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:15.924676 15264 server_base.cc:1034] running on GCE node
I20260809 04:39:15.925508 15264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:15.926321 15264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:15.927507 15264 hybrid_clock.cc:648] HybridClock initialized: now 1786250355927454 us; error 45 us; skew 500 ppm
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:15.930378 15264 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:39:15.931301 15264 webserver.cc:460] Webserver started at http://127.19.92.3:33773/ using document root <none> and password file <none>
I20260809 04:39:15.931850 15264 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:15.931910 15264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:15.932106 15264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:15.934258 15264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "ae1305f98f4f4ff38f458467bcd98d67"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "1d7b505d5ba00337771fed7653ce5e98"
server_key_iv: "c9f3c20a574b425772a47d933719185f"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.934795 15264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "ae1305f98f4f4ff38f458467bcd98d67"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "1d7b505d5ba00337771fed7653ce5e98"
server_key_iv: "c9f3c20a574b425772a47d933719185f"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.938727 15264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:39:15.941358 15289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.942698 15264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:15.942811 15264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "ae1305f98f4f4ff38f458467bcd98d67"
format_stamp: "Formatted at 2026-08-09 04:39:15 on dist-test-slave-rmhg"
server_key: "1d7b505d5ba00337771fed7653ce5e98"
server_key_iv: "c9f3c20a574b425772a47d933719185f"
server_key_version: "encryptionkey@0"
I20260809 04:39:15.942932 15264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:15.953889 15264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:15.957626 15264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:15.957834 15264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:15.958504 15264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:15.959524 15264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:15.959574 15264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.959616 15264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:15.959647 15264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:15.968389 15264 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:35717
I20260809 04:39:15.968427 15450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:35717 every 8 connection(s)
I20260809 04:39:15.969424 15264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
Aug 09 04:39:15 dist-test-slave-rmhg krb5kdc[14709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250355, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:15.979116 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 15264
I20260809 04:39:15.979246 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:39:15.979542 19824 external_mini_cluster.cc:1428] Setting key 37517a77718a291d5d35c75c79e474b2
I20260809 04:39:15.982910 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:15.971317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:33849 (local address 127.19.92.62:34545)
0809 04:39:15.971446 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:15.971449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:15.972339 (+ 890us) server_negotiation.cc:407] Connection header received
0809 04:39:15.973692 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:15.973695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:15.973756 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:15.973829 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:15.975018 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.975553 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.976687 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.976903 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.976990 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:15.979483 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:15.979495 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:15.979498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:15.979528 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:15.980977 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:15.981552 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:15.981555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:15.981557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:15.981650 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:15.981986 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:15.981988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:15.981989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:15.982171 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:15.982243 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:15.982694 (+ 451us) server_negotiation.cc:299] Negotiation successful
0809 04:39:15.982792 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:39:15.983719 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:15.971722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34545 (local address 127.19.92.3:33849)
0809 04:39:15.972189 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:15.972231 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:15.973476 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:15.974022 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:15.974029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:15.974382 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:15.974866 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:15.974880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.975672 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.975675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:15.976563 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:15.976573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.977050 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.977054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:15.977235 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:15.977791 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:15.977808 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:15.979287 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:15.981080 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:15.981086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:15.981101 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:15.981463 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:15.981734 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:15.981737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:15.981738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:15.981889 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:15.982250 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:15.982255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:15.982493 (+ 238us) client_negotiation.cc:769] Sending connection context
0809 04:39:15.982648 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 04:39:15.982901 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":140,"threads_started":1}
I20260809 04:39:15.984747 15451 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34545
I20260809 04:39:15.984967 15451 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:15.985489 15451 heartbeater.cc:511] Master 127.19.92.62:34545 requested a full tablet report, sending...
I20260809 04:39:15.986428 14773 ts_manager.cc:194] Registered new tserver with Master: ae1305f98f4f4ff38f458467bcd98d67 (127.19.92.3:35717)
I20260809 04:39:15.986984 14773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:33849
I20260809 04:39:15.993338 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:39:16.001979 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:15.995183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.19.92.62:34545)
0809 04:39:15.995289 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:15.995292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:15.995474 (+ 182us) server_negotiation.cc:407] Connection header received
0809 04:39:15.995575 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:15.995577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:15.995612 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:15.995672 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:15.996296 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.996792 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.997597 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:15.997803 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:15.997873 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:15.999273 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:15.999277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:15.999279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:15.999300 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:16.000431 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:16.000896 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:16.000898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:16.000900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:16.000978 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:16.001276 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:16.001279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:16.001280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:16.001476 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:16.001585 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.001777 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.001850 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:39:16.004856 14773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:39:16.007934 14773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:39:16.022287 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.016611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:34863 (local address 127.0.0.1:40110)
0809 04:39:16.017595 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:16.017608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:16.017677 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:16.018372 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:16.018375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:16.018393 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:16.018474 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.018480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.019506 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.019510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.020814 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.020825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.021740 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.021743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.021861 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.021901 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 04:39:16.022008 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 04:39:16.022133 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":94,"threads_started":1}
I20260809 04:39:16.022488 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.016466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.19.92.3:35717)
0809 04:39:16.017595 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.017608 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.017628 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:39:16.017700 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.017706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.017922 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.018424 (+ 502us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:16.018587 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.019693 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.021075 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.021903 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.022030 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.022180 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.022236 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.022335 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":135,"threads_started":1}
I20260809 04:39:16.022413 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.016301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:35717 (local address 127.0.0.1:49768)
0809 04:39:16.017184 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:16.017234 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:16.017366 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:16.018104 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:16.018110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:16.018135 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:16.018273 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.018278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.019826 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.019830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.020968 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.020975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.022040 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.022043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.022124 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.022145 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:39:16.022202 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 04:39:16.022255 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":48,"thread_start_us":70,"threads_started":1}
I20260809 04:39:16.023012 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.017144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.19.92.1:46291)
0809 04:39:16.018021 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.018035 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.018055 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:39:16.018115 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.018119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.018255 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.018394 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:16.018802 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.019874 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.021240 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.022015 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.022102 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.022214 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.022763 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.022876 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":97,"threads_started":1}
I20260809 04:39:16.022665 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.016343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:46291 (local address 127.0.0.1:45102)
0809 04:39:16.017402 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:16.017416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:16.017509 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:16.018571 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:16.018574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:16.018592 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:16.018682 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.018688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.019999 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.020002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.021110 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.021121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.022276 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.022279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.022362 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.022385 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:39:16.022468 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 04:39:16.022540 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":21,"thread_start_us":110,"threads_started":1}
I20260809 04:39:16.024554 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.017111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.19.92.2:34863)
0809 04:39:16.017914 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.017927 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.017950 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:39:16.018026 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.018032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.018226 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.018394 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:16.018572 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.019374 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.020924 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.021628 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.021724 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.021882 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.022790 (+ 908us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.022875 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":142,"threads_started":1}
I20260809 04:39:16.026005 15365 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff45e084a1142059d5b842ce2e1917a (DEFAULT_TABLE table=test-table [id=ff5bd2b25bb64baaa15fa07545f41792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:16.026505 14958 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff45e084a1142059d5b842ce2e1917a (DEFAULT_TABLE table=test-table [id=ff5bd2b25bb64baaa15fa07545f41792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:16.027189 15365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff45e084a1142059d5b842ce2e1917a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:16.027572 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff45e084a1142059d5b842ce2e1917a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:16.027292 15165 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff45e084a1142059d5b842ce2e1917a (DEFAULT_TABLE table=test-table [id=ff5bd2b25bb64baaa15fa07545f41792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:16.028260 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff45e084a1142059d5b842ce2e1917a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:16.033488 15479 tablet_bootstrap.cc:492] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: Bootstrap starting.
I20260809 04:39:16.034870 15480 tablet_bootstrap.cc:492] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea: Bootstrap starting.
I20260809 04:39:16.035257 15481 tablet_bootstrap.cc:492] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: Bootstrap starting.
I20260809 04:39:16.035851 15479 tablet_bootstrap.cc:654] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:16.037207 15479 log.cc:824] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:16.037837 15481 tablet_bootstrap.cc:654] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:16.038695 15481 log.cc:824] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:16.039201 15480 tablet_bootstrap.cc:654] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:16.039525 15479 tablet_bootstrap.cc:492] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: No bootstrap required, opened a new log
I20260809 04:39:16.039741 15479 ts_tablet_manager.cc:1397] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:39:16.040385 15480 log.cc:824] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:16.040473 15481 tablet_bootstrap.cc:492] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: No bootstrap required, opened a new log
I20260809 04:39:16.040668 15481 ts_tablet_manager.cc:1397] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:39:16.042455 15480 tablet_bootstrap.cc:492] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea: No bootstrap required, opened a new log
I20260809 04:39:16.042652 15480 ts_tablet_manager.cc:1397] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260809 04:39:16.043695 15481 raft_consensus.cc:348] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.043677 15479 raft_consensus.cc:348] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.043953 15481 raft_consensus.cc:374] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:16.043979 15479 raft_consensus.cc:374] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:16.044029 15481 raft_consensus.cc:729] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0977ce174f3b44dda924b29cf2daf681, State: Initialized, Role: FOLLOWER
I20260809 04:39:16.044070 15479 raft_consensus.cc:729] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1305f98f4f4ff38f458467bcd98d67, State: Initialized, Role: FOLLOWER
I20260809 04:39:16.044552 15481 consensus_queue.cc:260] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.044600 15479 consensus_queue.cc:260] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.045657 15481 ts_tablet_manager.cc:1428] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 04:39:16.045727 15479 ts_tablet_manager.cc:1428] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 04:39:16.046093 15259 heartbeater.cc:503] Master 127.19.92.62:34545 was elected leader, sending a full tablet report...
I20260809 04:39:16.046274 15451 heartbeater.cc:503] Master 127.19.92.62:34545 was elected leader, sending a full tablet report...
I20260809 04:39:16.047679 15480 raft_consensus.cc:348] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.047930 15480 raft_consensus.cc:374] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:16.047988 15480 raft_consensus.cc:729] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbac262fbe247a38e9eaea6e1ccdaea, State: Initialized, Role: FOLLOWER
I20260809 04:39:16.048401 15480 consensus_queue.cc:260] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.049003 15066 heartbeater.cc:503] Master 127.19.92.62:34545 was elected leader, sending a full tablet report...
I20260809 04:39:16.050294 15480 ts_tablet_manager.cc:1428] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260809 04:39:16.119783 15486 raft_consensus.cc:482] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:16.119992 15486 raft_consensus.cc:504] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.121243 15486 leader_election.cc:288] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae1305f98f4f4ff38f458467bcd98d67 (127.19.92.3:35717), ebbac262fbe247a38e9eaea6e1ccdaea (127.19.92.1:46291)
I20260809 04:39:16.126621 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.121984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:41201 (local address 127.19.92.1:46291)
0809 04:39:16.122128 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.122133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.122146 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:39:16.122196 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.122199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.122243 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.122316 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:16.122609 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.123412 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.125250 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.125896 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.126040 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.126125 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.126399 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.126482 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:39:16.126643 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.121476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:46291 (local address 127.19.92.2:41201)
0809 04:39:16.121676 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:16.121694 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:16.121806 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:16.122345 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:16.122347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:16.122360 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:16.122478 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.122485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.123562 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.123567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.125097 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.125111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.126043 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.126047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.126248 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.126290 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 04:39:16.126369 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 04:39:16.126460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":132}
I20260809 04:39:16.126709 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.121576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:42047 (local address 127.19.92.3:35717)
0809 04:39:16.121857 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.121861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.121924 (+ 63us) server_negotiation.cc:407] Connection header received
0809 04:39:16.122049 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.122052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.122110 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.122203 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:16.122609 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.123412 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.125251 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.125947 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.126046 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.126117 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.126526 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.126604 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260809 04:39:16.126652 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.121473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:35717 (local address 127.19.92.2:42047)
0809 04:39:16.121854 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:16.121867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:16.121955 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:16.122252 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:16.122255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:16.122277 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:16.122418 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.122427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.123562 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.123567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.125099 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.125111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.126058 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.126061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.126253 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.126364 (+ 111us) client_negotiation.cc:769] Sending connection context
0809 04:39:16.126477 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 04:39:16.126541 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":236,"thread_start_us":134,"threads_started":1}
I20260809 04:39:16.127467 14993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff45e084a1142059d5b842ce2e1917a" candidate_uuid: "0977ce174f3b44dda924b29cf2daf681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" is_pre_election: true
I20260809 04:39:16.127769 14993 raft_consensus.cc:2454] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0977ce174f3b44dda924b29cf2daf681 in term 0.
I20260809 04:39:16.128273 15093 leader_election.cc:302] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0977ce174f3b44dda924b29cf2daf681, ebbac262fbe247a38e9eaea6e1ccdaea; no voters:
I20260809 04:39:16.128598 15486 raft_consensus.cc:2781] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:39:16.128579 15393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff45e084a1142059d5b842ce2e1917a" candidate_uuid: "0977ce174f3b44dda924b29cf2daf681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1305f98f4f4ff38f458467bcd98d67" is_pre_election: true
I20260809 04:39:16.128686 15486 raft_consensus.cc:482] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:39:16.128742 15486 raft_consensus.cc:3037] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:16.128854 15393 raft_consensus.cc:2454] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0977ce174f3b44dda924b29cf2daf681 in term 0.
I20260809 04:39:16.129940 15486 raft_consensus.cc:504] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.130290 15486 leader_election.cc:288] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [CANDIDATE]: Term 1 election: Requested vote from peers ae1305f98f4f4ff38f458467bcd98d67 (127.19.92.3:35717), ebbac262fbe247a38e9eaea6e1ccdaea (127.19.92.1:46291)
I20260809 04:39:16.130640 14993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff45e084a1142059d5b842ce2e1917a" candidate_uuid: "0977ce174f3b44dda924b29cf2daf681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea"
I20260809 04:39:16.130729 15393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff45e084a1142059d5b842ce2e1917a" candidate_uuid: "0977ce174f3b44dda924b29cf2daf681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1305f98f4f4ff38f458467bcd98d67"
I20260809 04:39:16.130775 14993 raft_consensus.cc:3037] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:16.130837 15393 raft_consensus.cc:3037] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:16.132123 14993 raft_consensus.cc:2454] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0977ce174f3b44dda924b29cf2daf681 in term 1.
I20260809 04:39:16.132123 15393 raft_consensus.cc:2454] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0977ce174f3b44dda924b29cf2daf681 in term 1.
I20260809 04:39:16.132454 15093 leader_election.cc:302] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0977ce174f3b44dda924b29cf2daf681, ebbac262fbe247a38e9eaea6e1ccdaea; no voters:
I20260809 04:39:16.132632 15486 raft_consensus.cc:2781] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:16.132884 15486 raft_consensus.cc:686] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 1 LEADER]: Becoming Leader. State: Replica: 0977ce174f3b44dda924b29cf2daf681, State: Running, Role: LEADER
I20260809 04:39:16.133272 15486 consensus_queue.cc:237] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } }
I20260809 04:39:16.136704 14778 catalog_manager.cc:5495] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0977ce174f3b44dda924b29cf2daf681 (127.19.92.2). New cstate: current_term: 1 leader_uuid: "0977ce174f3b44dda924b29cf2daf681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0977ce174f3b44dda924b29cf2daf681" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 34863 } health_report { overall_health: HEALTHY } } }
I20260809 04:39:16.176642 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.173469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40116 (local address 127.19.92.2:34863)
0809 04:39:16.173595 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.173599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.173613 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:39:16.173708 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.173710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.173767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.173859 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:16.174054 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.174550 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.175322 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.175539 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.175612 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.175851 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:16.176278 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:16.176388 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.176491 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.176529 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 04:39:16.182658 14993 raft_consensus.cc:1261] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea [term 1 FOLLOWER]: Refusing update from remote peer 0977ce174f3b44dda924b29cf2daf681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:16.182653 15393 raft_consensus.cc:1261] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 1 FOLLOWER]: Refusing update from remote peer 0977ce174f3b44dda924b29cf2daf681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:16.183306 15490 consensus_queue.cc:1035] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbac262fbe247a38e9eaea6e1ccdaea" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:16.183486 15486 consensus_queue.cc:1035] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1305f98f4f4ff38f458467bcd98d67" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:16.190441 15495 mvcc.cc:205] Tried to move back new op lower bound from 7316481458916708352 to 7316481458724614144. Current Snapshot: MvccSnapshot[applied={T|T < 7316481458916708352}]
I20260809 04:39:16.190907 15493 mvcc.cc:205] Tried to move back new op lower bound from 7316481458916708352 to 7316481458724614144. Current Snapshot: MvccSnapshot[applied={T|T < 7316481458916708352}]
I20260809 04:39:16.193858 15496 mvcc.cc:205] Tried to move back new op lower bound from 7316481458916708352 to 7316481458724614144. Current Snapshot: MvccSnapshot[applied={T|T < 7316481458916708352}]
I20260809 04:39:16.198642 14778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:16.198797 14778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:16.201117 14778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b766b0c35bf54fe8a0f9547600bd57c2: Sending DeleteTablet for 3 replicas of tablet 4ff45e084a1142059d5b842ce2e1917a
I20260809 04:39:16.201961 14958 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff45e084a1142059d5b842ce2e1917a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:16 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:45102
I20260809 04:39:16.201959 15365 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff45e084a1142059d5b842ce2e1917a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:16 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:49768
I20260809 04:39:16.202538 15508 tablet_replica.cc:330] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: stopping tablet replica
I20260809 04:39:16.202670 15165 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff45e084a1142059d5b842ce2e1917a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:16 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:40110
I20260809 04:39:16.202947 15508 raft_consensus.cc:2229] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:16.203315 15508 raft_consensus.cc:2258] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:16.203449 15511 tablet_replica.cc:330] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: stopping tablet replica
I20260809 04:39:16.203541 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 14873
I20260809 04:39:16.203747 15507 tablet_replica.cc:330] T 4ff45e084a1142059d5b842ce2e1917a P ebbac262fbe247a38e9eaea6e1ccdaea: stopping tablet replica
I20260809 04:39:16.203769 15511 raft_consensus.cc:2229] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:39:16.204137 15511 raft_consensus.cc:2258] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:16.207592 15508 ts_tablet_manager.cc:1905] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:16.210631 15508 ts_tablet_manager.cc:1918] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:16.210721 15508 log.cc:1196] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/4ff45e084a1142059d5b842ce2e1917a
I20260809 04:39:16.211000 15508 ts_tablet_manager.cc:1939] T 4ff45e084a1142059d5b842ce2e1917a P ae1305f98f4f4ff38f458467bcd98d67: Deleting consensus metadata
I20260809 04:39:16.212323 14760 catalog_manager.cc:4841] TS ae1305f98f4f4ff38f458467bcd98d67 (127.19.92.3:35717): tablet 4ff45e084a1142059d5b842ce2e1917a (table test-table [id=ff5bd2b25bb64baaa15fa07545f41792]) successfully deleted
I20260809 04:39:16.213608 15511 ts_tablet_manager.cc:1905] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:16.215966 15511 ts_tablet_manager.cc:1918] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:16.216073 15511 log.cc:1196] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/4ff45e084a1142059d5b842ce2e1917a
I20260809 04:39:16.216360 15511 ts_tablet_manager.cc:1939] T 4ff45e084a1142059d5b842ce2e1917a P 0977ce174f3b44dda924b29cf2daf681: Deleting consensus metadata
I20260809 04:39:16.217612 14758 catalog_manager.cc:4841] TS 0977ce174f3b44dda924b29cf2daf681 (127.19.92.2:34863): tablet 4ff45e084a1142059d5b842ce2e1917a (table test-table [id=ff5bd2b25bb64baaa15fa07545f41792]) successfully deleted
W20260809 04:39:16.222321 14761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:46291 (error 108)
I20260809 04:39:16.223305 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 15072
I20260809 04:39:16.223452 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.223139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:46291 (local address 127.0.0.1:45116)
0809 04:39:16.223273 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:16.223346 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:39:16.229605 14761 catalog_manager.cc:4584] TS ebbac262fbe247a38e9eaea6e1ccdaea (127.19.92.1:46291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ff45e084a1142059d5b842ce2e1917a: Network error: Client connection negotiation failed: client connection to 127.19.92.1:46291: connect: Connection refused (error 111)
I20260809 04:39:16.238206 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 15264
I20260809 04:39:16.255048 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 14734
2026-08-09T04:39:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2449 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T04:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:16Z Disabled control of system clock
I20260809 04:39:16.295105 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:34277
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:34277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:39:16.413245 15521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:16.413528 15521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:16.413570 15521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:16.417635 15521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:16.417699 15521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:16.417713 15521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:16.417729 15521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:16.417747 15521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:16.422542 15521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:34277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:34277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:16.423645 15521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:16.425062 15521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:16.433595 15527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:16.433558 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:16.433558 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:16.433955 15521 server_base.cc:1034] running on GCE node
I20260809 04:39:16.434494 15521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:16.435593 15521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:16.436779 15521 hybrid_clock.cc:648] HybridClock initialized: now 1786250356436759 us; error 35 us; skew 500 ppm
I20260809 04:39:16.438899 15521 webserver.cc:460] Webserver started at http://127.19.92.62:33347/ using document root <none> and password file <none>
I20260809 04:39:16.439496 15521 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:16.439570 15521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:16.439781 15521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:16.442024 15521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "05006447f8fb460592d83d474a102b36"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "474f1c3bb0a88783ea042b6e67464e41"
server_key_iv: "732b96f8afe6969f70a2a4440dddf948"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.442602 15521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "05006447f8fb460592d83d474a102b36"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "474f1c3bb0a88783ea042b6e67464e41"
server_key_iv: "732b96f8afe6969f70a2a4440dddf948"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.446656 15521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:39:16.449221 15539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:16.450511 15521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:16.450649 15521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "05006447f8fb460592d83d474a102b36"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "474f1c3bb0a88783ea042b6e67464e41"
server_key_iv: "732b96f8afe6969f70a2a4440dddf948"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.450752 15521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:16.465943 15521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:16.466670 15521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:16.466847 15521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:16.473719 15521 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:34277
I20260809 04:39:16.473752 15607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:34277 every 8 connection(s)
I20260809 04:39:16.474669 15521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:16.477854 15612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:16.481518 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 15521
I20260809 04:39:16.481678 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:16.481999 19824 external_mini_cluster.cc:1428] Setting key 6d6536119a82ada9c02e01444d6c646b
I20260809 04:39:16.484364 15612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36: Bootstrap starting.
I20260809 04:39:16.486829 15612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:16.487636 15612 log.cc:824] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:16.489660 15612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36: No bootstrap required, opened a new log
I20260809 04:39:16.491746 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.483598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.19.92.62:34277)
0809 04:39:16.484355 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.484372 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.484426 (+ 54us) server_negotiation.cc:407] Connection header received
0809 04:39:16.485205 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.485233 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.485668 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.486011 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:16.486483 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.487553 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.488487 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.488979 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.489132 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.490002 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:16.490007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:16.490021 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:16.490422 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:16.490483 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:16.490502 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:16.490591 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.490748 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.491023 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":158,"threads_started":1}
I20260809 04:39:16.493402 15612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05006447f8fb460592d83d474a102b36" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34277 } }
I20260809 04:39:16.493645 15612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:16.493710 15612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05006447f8fb460592d83d474a102b36, State: Initialized, Role: FOLLOWER
I20260809 04:39:16.494196 15612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05006447f8fb460592d83d474a102b36" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34277 } }
I20260809 04:39:16.494349 15612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:16.494396 15612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:16.494472 15612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:16.495360 15612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05006447f8fb460592d83d474a102b36" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34277 } }
I20260809 04:39:16.495671 15612 leader_election.cc:302] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05006447f8fb460592d83d474a102b36; no voters:
I20260809 04:39:16.495950 15612 leader_election.cc:288] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:16.496145 15620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:16.496390 15620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [term 1 LEADER]: Becoming Leader. State: Replica: 05006447f8fb460592d83d474a102b36, State: Running, Role: LEADER
I20260809 04:39:16.496752 15620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05006447f8fb460592d83d474a102b36" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34277 } }
I20260809 04:39:16.497118 15612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:16.498236 15622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05006447f8fb460592d83d474a102b36. Latest consensus state: current_term: 1 leader_uuid: "05006447f8fb460592d83d474a102b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05006447f8fb460592d83d474a102b36" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34277 } } }
I20260809 04:39:16.498359 15622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:16.498624 15621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05006447f8fb460592d83d474a102b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05006447f8fb460592d83d474a102b36" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 34277 } } }
I20260809 04:39:16.498772 15630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:16.498852 15621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:16.501761 15630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:16.506641 15630 catalog_manager.cc:1347] Generated new cluster ID: 417d0b90d1514395a51b7d2ff9c1c22d
I20260809 04:39:16.506711 15630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:16.515481 15630 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:16.516719 15630 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:16.525921 15630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36: Generated new TSK 0
I20260809 04:39:16.526870 15630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:16.536288 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34277
--builtin_ntp_servers=127.19.92.20:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:39:16.650557 15644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:16.650857 15644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:16.650913 15644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:16.655015 15644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:16.655076 15644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:16.655153 15644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:39:16.659919 15644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:16.661012 15644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:16.662421 15644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:16.671763 15654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:16.671808 15655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:16.671783 15658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:16.671868 15644 server_base.cc:1034] running on GCE node
I20260809 04:39:16.672436 15644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:16.673023 15644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:16.674187 15644 hybrid_clock.cc:648] HybridClock initialized: now 1786250356674160 us; error 41 us; skew 500 ppm
I20260809 04:39:16.675947 15644 webserver.cc:460] Webserver started at http://127.19.92.1:44225/ using document root <none> and password file <none>
I20260809 04:39:16.676509 15644 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:16.676571 15644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:16.676759 15644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:16.678848 15644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "53764bbdcef0495baed9b6d7c9dc1302"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "865b9f6379ba7beb2f6d0889c5641e8d"
server_key_iv: "30aac3387945705857ceba23c7cc7d92"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.679407 15644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "53764bbdcef0495baed9b6d7c9dc1302"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "865b9f6379ba7beb2f6d0889c5641e8d"
server_key_iv: "30aac3387945705857ceba23c7cc7d92"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.683182 15644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 04:39:16.686014 15664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:16.687404 15644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:16.687527 15644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "53764bbdcef0495baed9b6d7c9dc1302"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "865b9f6379ba7beb2f6d0889c5641e8d"
server_key_iv: "30aac3387945705857ceba23c7cc7d92"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.687659 15644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:16.702138 15644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:16.702852 15644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:16.703039 15644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:16.703706 15644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:16.704735 15644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:16.704783 15644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:16.704828 15644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:16.704856 15644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:16.713481 15644 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:46599
I20260809 04:39:16.713512 15828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:46599 every 8 connection(s)
I20260809 04:39:16.714460 15644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:39:16.723658 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.716386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:60631 (local address 127.19.92.62:34277)
0809 04:39:16.716566 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.716571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.717631 (+ 1060us) server_negotiation.cc:407] Connection header received
0809 04:39:16.719028 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.719032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.719080 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.719155 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:16.720068 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.720618 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.721723 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.721955 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.722036 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.723120 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:16.723123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:16.723125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:16.723138 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:16.723159 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:16.723167 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:16.723222 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.723463 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.723529 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 04:39:16.724221 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 15644
I20260809 04:39:16.724360 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:39:16.724728 19824 external_mini_cluster.cc:1428] Setting key ac71b549539051c1054722a3ef4e34a7
I20260809 04:39:16.724876 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.716780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34277 (local address 127.19.92.1:60631)
0809 04:39:16.717480 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:16.717522 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:16.718806 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:16.719310 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:16.719317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:16.719404 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:16.719911 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.719941 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.720707 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.720709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.721619 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.721629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.722073 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.722076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.722219 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.722843 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:16.722844 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:16.722889 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:16.722957 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:16.723290 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:16.723300 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:16.723306 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:39:16.723430 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 04:39:16.723695 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260809 04:39:16.726256 15829 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34277
I20260809 04:39:16.726514 15829 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:16.727046 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34277
--builtin_ntp_servers=127.19.92.20:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:39:16.727072 15829 heartbeater.cc:511] Master 127.19.92.62:34277 requested a full tablet report, sending...
I20260809 04:39:16.728624 15558 ts_manager.cc:194] Registered new tserver with Master: 53764bbdcef0495baed9b6d7c9dc1302 (127.19.92.1:46599)
I20260809 04:39:16.729822 15558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.1:60631
W20260809 04:39:16.840583 15835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:16.840840 15835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:16.840878 15835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:16.844918 15835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:16.844996 15835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:16.845053 15835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:39:16.850051 15835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:16.851243 15835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:16.852680 15835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:16.862479 15845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:16.862483 15843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:16.862432 15848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:16.862521 15835 server_base.cc:1034] running on GCE node
I20260809 04:39:16.863116 15835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:16.863767 15835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:16.864939 15835 hybrid_clock.cc:648] HybridClock initialized: now 1786250356864918 us; error 32 us; skew 500 ppm
I20260809 04:39:16.866798 15835 webserver.cc:460] Webserver started at http://127.19.92.2:41365/ using document root <none> and password file <none>
I20260809 04:39:16.867343 15835 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:16.867401 15835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:16.867571 15835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:16.869622 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "e1659769e6e449c186ef20ddd9ee75ac"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "66cae46b13efe07992f96c4e374d4774"
server_key_iv: "fb1cd69b686befee585183faf9d796de"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.870132 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "e1659769e6e449c186ef20ddd9ee75ac"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "66cae46b13efe07992f96c4e374d4774"
server_key_iv: "fb1cd69b686befee585183faf9d796de"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.873651 15835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:39:16.875994 15856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:16.877219 15835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:16.877326 15835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "e1659769e6e449c186ef20ddd9ee75ac"
format_stamp: "Formatted at 2026-08-09 04:39:16 on dist-test-slave-rmhg"
server_key: "66cae46b13efe07992f96c4e374d4774"
server_key_iv: "fb1cd69b686befee585183faf9d796de"
server_key_version: "encryptionkey@0"
I20260809 04:39:16.877421 15835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:16.894400 15835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:16.895141 15835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:16.895319 15835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:16.895901 15835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:16.896850 15835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:16.896896 15835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:16.896953 15835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:16.896983 15835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:16.905246 15835 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:44475
I20260809 04:39:16.905268 16024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:44475 every 8 connection(s)
I20260809 04:39:16.906126 15835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:39:16.914697 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 15835
I20260809 04:39:16.914845 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:39:16.914886 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.907876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:46487 (local address 127.19.92.62:34277)
0809 04:39:16.908015 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:16.908018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:16.909073 (+ 1055us) server_negotiation.cc:407] Connection header received
0809 04:39:16.910441 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:16.910444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:16.910494 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:16.910566 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:16.911393 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.911967 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.913087 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.913364 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.913446 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.914431 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:16.914435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:16.914438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:16.914451 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:16.914471 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:16.914479 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:16.914546 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:16.914671 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 04:39:16.914733 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:39:16.915206 19824 external_mini_cluster.cc:1428] Setting key 4ce0ce4139c5ca53b8d346641d676d5e
I20260809 04:39:16.915803 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:16.908260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34277 (local address 127.19.92.2:46487)
0809 04:39:16.908931 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:16.908971 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:16.910255 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:16.910718 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:16.910724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:16.910809 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:16.911288 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.911300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.912085 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.912088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.912982 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:16.912993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:16.913478 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:16.913481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:16.913626 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:16.914202 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:16.914203 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:16.914243 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:16.914308 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:16.914557 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:16.914562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:16.914566 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:39:16.914653 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 04:39:16.914905 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260809 04:39:16.916989 16025 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34277
I20260809 04:39:16.917279 16025 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:16.917539 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34277
--builtin_ntp_servers=127.19.92.20:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:39:16.917904 16025 heartbeater.cc:511] Master 127.19.92.62:34277 requested a full tablet report, sending...
I20260809 04:39:16.919073 15558 ts_manager.cc:194] Registered new tserver with Master: e1659769e6e449c186ef20ddd9ee75ac (127.19.92.2:44475)
I20260809 04:39:16.919867 15558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.2:46487
W20260809 04:39:17.036047 16033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:17.036321 16033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:17.036379 16033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:17.040482 16033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:17.040542 16033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:17.040617 16033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:39:17.045650 16033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:34277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:17.046867 16033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:17.048391 16033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:17.058205 16042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:17.058214 16039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:17.058264 16033 server_base.cc:1034] running on GCE node
W20260809 04:39:17.058200 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:17.059348 16033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:17.059979 16033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:17.061144 16033 hybrid_clock.cc:648] HybridClock initialized: now 1786250357061123 us; error 34 us; skew 500 ppm
I20260809 04:39:17.063000 16033 webserver.cc:460] Webserver started at http://127.19.92.3:39461/ using document root <none> and password file <none>
I20260809 04:39:17.063576 16033 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:17.063642 16033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:17.063832 16033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:17.065960 16033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "9c9ffce5989e449ca1395abd66f3993d"
format_stamp: "Formatted at 2026-08-09 04:39:17 on dist-test-slave-rmhg"
server_key: "23ddf612563a42fc146d4a28c9d8bcdf"
server_key_iv: "3fd570158994c8c61942084aba2d0fc4"
server_key_version: "encryptionkey@0"
I20260809 04:39:17.066507 16033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "9c9ffce5989e449ca1395abd66f3993d"
format_stamp: "Formatted at 2026-08-09 04:39:17 on dist-test-slave-rmhg"
server_key: "23ddf612563a42fc146d4a28c9d8bcdf"
server_key_iv: "3fd570158994c8c61942084aba2d0fc4"
server_key_version: "encryptionkey@0"
I20260809 04:39:17.070175 16033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:39:17.072579 16050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:17.073803 16033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:39:17.073911 16033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "9c9ffce5989e449ca1395abd66f3993d"
format_stamp: "Formatted at 2026-08-09 04:39:17 on dist-test-slave-rmhg"
server_key: "23ddf612563a42fc146d4a28c9d8bcdf"
server_key_iv: "3fd570158994c8c61942084aba2d0fc4"
server_key_version: "encryptionkey@0"
I20260809 04:39:17.074021 16033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:17.110529 16033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:17.111284 16033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:17.111470 16033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:17.112109 16033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:17.113112 16033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:17.113180 16033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:17.113242 16033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:17.113267 16033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:17.122624 16033 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:39871
I20260809 04:39:17.122645 16233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:39871 every 8 connection(s)
I20260809 04:39:17.123626 16033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:39:17.126911 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 16033
I20260809 04:39:17.127076 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:39:17.127398 19824 external_mini_cluster.cc:1428] Setting key 09f7dc387c1068d63e476002e3f296f5
I20260809 04:39:17.133626 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.125674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:39921 (local address 127.19.92.62:34277)
0809 04:39:17.125838 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:17.125842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:17.127138 (+ 1296us) server_negotiation.cc:407] Connection header received
0809 04:39:17.128889 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:17.128892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:17.128954 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:17.129027 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:17.130089 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.130616 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.131772 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.132001 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.132090 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.133127 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:17.133131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:17.133134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:17.133168 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:17.133191 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:17.133201 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:17.133265 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:17.133407 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 04:39:17.133482 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:39:17.134510 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.126120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:34277 (local address 127.19.92.3:39921)
0809 04:39:17.126951 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:17.127014 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:17.128670 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:17.129262 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:17.129269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:17.129394 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:17.129936 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.129952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.130735 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.130739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.131659 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.131669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.132138 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.132142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.132323 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.132876 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:17.132877 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:17.132918 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:17.132988 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:17.133275 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:17.133284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:17.133290 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:39:17.133403 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 04:39:17.133744 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":130,"threads_started":1}
I20260809 04:39:17.135547 16236 heartbeater.cc:347] Connected to a master server at 127.19.92.62:34277
I20260809 04:39:17.135756 16236 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:17.136234 16236 heartbeater.cc:511] Master 127.19.92.62:34277 requested a full tablet report, sending...
I20260809 04:39:17.137251 15558 ts_manager.cc:194] Registered new tserver with Master: 9c9ffce5989e449ca1395abd66f3993d (127.19.92.3:39871)
I20260809 04:39:17.137777 15558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.3:39921
I20260809 04:39:17.141049 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:39:17.146876 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.143050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.19.92.62:34277)
0809 04:39:17.143177 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:17.143181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:17.143369 (+ 188us) server_negotiation.cc:407] Connection header received
0809 04:39:17.143488 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:17.143491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:17.143538 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:17.143602 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:17.143873 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.144384 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.145242 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.145473 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.145561 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.146386 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:17.146390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:17.146392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:17.146404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:17.146422 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:17.146430 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:17.146534 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:17.146690 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 04:39:17.146730 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:39:17.149834 15558 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:39:17.153080 15558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:39:17.167944 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.161909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:46599 (local address 127.0.0.1:41836)
0809 04:39:17.163402 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:17.163415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:17.163523 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:17.163996 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:17.163998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:17.164015 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:17.164096 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.164101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.165429 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.165432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.166581 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.166591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.167374 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.167377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.167486 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.167536 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 04:39:17.167655 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 04:39:17.167785 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429,"spinlock_wait_cycles":768,"thread_start_us":93,"threads_started":1}
I20260809 04:39:17.167949 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.162434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.19.92.1:46599)
0809 04:39:17.163260 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:17.163272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:17.163615 (+ 343us) server_negotiation.cc:407] Connection header received
0809 04:39:17.163682 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:17.163686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:17.163866 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:17.164010 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:17.164173 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.165294 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.166695 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.167254 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.167352 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.167564 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:17.167686 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 04:39:17.167785 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":129,"threads_started":1}
I20260809 04:39:17.168159 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.161616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:44475 (local address 127.0.0.1:59100)
0809 04:39:17.162443 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:17.162498 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:17.162701 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:17.163421 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:17.163427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:17.163453 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:17.163657 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.163662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.165190 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.165195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.166410 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.166420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.167737 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.167740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.167835 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.167859 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:39:17.167923 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 04:39:17.168009 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":52,"spinlock_wait_cycles":45440,"thread_start_us":83,"threads_started":1}
I20260809 04:39:17.168531 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.161738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.19.92.2:44475)
0809 04:39:17.162198 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:17.162211 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:17.162590 (+ 379us) server_negotiation.cc:407] Connection header received
0809 04:39:17.162903 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:17.162910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:17.163144 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:17.163252 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:17.163770 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.165029 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.166552 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.167385 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.167488 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.167675 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:17.168267 (+ 592us) server_negotiation.cc:299] Negotiation successful
0809 04:39:17.168370 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":142,"threads_started":1}
I20260809 04:39:17.169586 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.161710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:39871 (local address 127.0.0.1:38832)
0809 04:39:17.162864 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:17.162879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:17.162980 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:17.163432 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:17.163435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:17.163460 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:17.163645 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.163653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.166513 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.166516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.167608 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.167620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.169074 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.169078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.169197 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.169220 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:39:17.169300 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 04:39:17.169369 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":3,"spinlock_wait_cycles":36480,"thread_start_us":112,"threads_started":1}
I20260809 04:39:17.170137 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.162434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.19.92.3:39871)
0809 04:39:17.162858 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:17.162870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:17.162972 (+ 102us) server_negotiation.cc:407] Connection header received
0809 04:39:17.163094 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:17.163100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:17.163296 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:17.163458 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:17.165205 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.166394 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.167750 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.168730 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.168836 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.169008 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:17.169866 (+ 858us) server_negotiation.cc:299] Negotiation successful
0809 04:39:17.169972 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":105,"threads_started":1}
I20260809 04:39:17.171399 15932 tablet_service.cc:1467] Processing CreateTablet for tablet b88ed7d217204e608d9354c398d34629 (DEFAULT_TABLE table=test-table [id=7a45be60f159438f947928970e1d2601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:17.171411 15736 tablet_service.cc:1467] Processing CreateTablet for tablet b88ed7d217204e608d9354c398d34629 (DEFAULT_TABLE table=test-table [id=7a45be60f159438f947928970e1d2601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:17.172395 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ed7d217204e608d9354c398d34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:17.172456 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ed7d217204e608d9354c398d34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:17.172801 16133 tablet_service.cc:1467] Processing CreateTablet for tablet b88ed7d217204e608d9354c398d34629 (DEFAULT_TABLE table=test-table [id=7a45be60f159438f947928970e1d2601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:17.173770 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ed7d217204e608d9354c398d34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:17.178923 16262 tablet_bootstrap.cc:492] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Bootstrap starting.
I20260809 04:39:17.180397 16263 tablet_bootstrap.cc:492] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302: Bootstrap starting.
I20260809 04:39:17.181449 16264 tablet_bootstrap.cc:492] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Bootstrap starting.
I20260809 04:39:17.181502 16262 tablet_bootstrap.cc:654] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:17.182387 16262 log.cc:824] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:17.182729 16263 tablet_bootstrap.cc:654] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:17.183091 16264 tablet_bootstrap.cc:654] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:17.183583 16263 log.cc:824] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:17.183863 16264 log.cc:824] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:17.184645 16262 tablet_bootstrap.cc:492] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: No bootstrap required, opened a new log
I20260809 04:39:17.184839 16262 ts_tablet_manager.cc:1397] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:39:17.185817 16263 tablet_bootstrap.cc:492] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302: No bootstrap required, opened a new log
I20260809 04:39:17.185822 16264 tablet_bootstrap.cc:492] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: No bootstrap required, opened a new log
I20260809 04:39:17.186009 16264 ts_tablet_manager.cc:1397] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 04:39:17.186009 16263 ts_tablet_manager.cc:1397] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260809 04:39:17.189042 16264 raft_consensus.cc:348] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.189093 16262 raft_consensus.cc:348] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.189303 16264 raft_consensus.cc:374] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:17.189366 16264 raft_consensus.cc:729] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c9ffce5989e449ca1395abd66f3993d, State: Initialized, Role: FOLLOWER
I20260809 04:39:17.189399 16262 raft_consensus.cc:374] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:17.189461 16262 raft_consensus.cc:729] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1659769e6e449c186ef20ddd9ee75ac, State: Initialized, Role: FOLLOWER
I20260809 04:39:17.189836 16264 consensus_queue.cc:260] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.189926 16262 consensus_queue.cc:260] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.190225 16263 raft_consensus.cc:348] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.190510 16263 raft_consensus.cc:374] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:17.190600 16263 raft_consensus.cc:729] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53764bbdcef0495baed9b6d7c9dc1302, State: Initialized, Role: FOLLOWER
I20260809 04:39:17.190892 16262 ts_tablet_manager.cc:1428] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260809 04:39:17.190951 16264 ts_tablet_manager.cc:1428] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 04:39:17.191037 16263 consensus_queue.cc:260] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.191323 16025 heartbeater.cc:503] Master 127.19.92.62:34277 was elected leader, sending a full tablet report...
I20260809 04:39:17.191603 16236 heartbeater.cc:503] Master 127.19.92.62:34277 was elected leader, sending a full tablet report...
I20260809 04:39:17.191773 15829 heartbeater.cc:503] Master 127.19.92.62:34277 was elected leader, sending a full tablet report...
I20260809 04:39:17.191957 16263 ts_tablet_manager.cc:1428] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
W20260809 04:39:17.215642 15832 tablet.cc:2369] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:17.341527 16272 raft_consensus.cc:482] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:17.341794 16272 raft_consensus.cc:504] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.343120 16272 leader_election.cc:288] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1659769e6e449c186ef20ddd9ee75ac (127.19.92.2:44475), 53764bbdcef0495baed9b6d7c9dc1302 (127.19.92.1:46599)
I20260809 04:39:17.347162 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.343544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:56721 (local address 127.19.92.2:44475)
0809 04:39:17.343675 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:17.343680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:17.343699 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:39:17.343763 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:17.343767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:17.343818 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:17.343925 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:17.344222 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.344908 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.346123 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.346593 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.346675 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.346730 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:17.346944 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 04:39:17.347032 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:39:17.347147 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.343431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:44475 (local address 127.19.92.3:56721)
0809 04:39:17.343558 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:17.343574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:17.343679 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:17.343949 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:17.343952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:17.343979 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:17.344088 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.344094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.345038 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.345041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.345992 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.346003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.346715 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.346719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.346818 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.346845 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 04:39:17.346918 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 04:39:17.347001 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":5}
I20260809 04:39:17.347267 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.343617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:56979 (local address 127.19.92.1:46599)
0809 04:39:17.343729 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:17.343733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:17.344169 (+ 436us) server_negotiation.cc:407] Connection header received
0809 04:39:17.344248 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:17.344250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:17.344296 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:17.344351 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:17.344661 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.345258 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.346302 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.346731 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.346830 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.346886 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:17.347086 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 04:39:17.347160 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 04:39:17.347308 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.343431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:46599 (local address 127.19.92.3:56979)
0809 04:39:17.344065 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:17.344076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:17.344167 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:17.344374 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:17.344377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:17.344395 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:17.344555 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.344562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.345360 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.345363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.346175 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:17.346187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.346849 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.346852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:17.346957 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.346981 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:39:17.347055 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:39:17.347122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":182,"thread_start_us":168,"threads_started":1}
I20260809 04:39:17.347913 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ed7d217204e608d9354c398d34629" candidate_uuid: "9c9ffce5989e449ca1395abd66f3993d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1659769e6e449c186ef20ddd9ee75ac" is_pre_election: true
I20260809 04:39:17.347944 15767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ed7d217204e608d9354c398d34629" candidate_uuid: "9c9ffce5989e449ca1395abd66f3993d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53764bbdcef0495baed9b6d7c9dc1302" is_pre_election: true
I20260809 04:39:17.348225 15963 raft_consensus.cc:2454] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c9ffce5989e449ca1395abd66f3993d in term 0.
I20260809 04:39:17.348232 15767 raft_consensus.cc:2454] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c9ffce5989e449ca1395abd66f3993d in term 0.
I20260809 04:39:17.348708 16053 leader_election.cc:302] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53764bbdcef0495baed9b6d7c9dc1302, 9c9ffce5989e449ca1395abd66f3993d; no voters:
I20260809 04:39:17.349041 16272 raft_consensus.cc:2781] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:39:17.349169 16272 raft_consensus.cc:482] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:39:17.349227 16272 raft_consensus.cc:3037] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:17.350315 16272 raft_consensus.cc:504] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.350675 16272 leader_election.cc:288] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [CANDIDATE]: Term 1 election: Requested vote from peers e1659769e6e449c186ef20ddd9ee75ac (127.19.92.2:44475), 53764bbdcef0495baed9b6d7c9dc1302 (127.19.92.1:46599)
I20260809 04:39:17.351027 15767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ed7d217204e608d9354c398d34629" candidate_uuid: "9c9ffce5989e449ca1395abd66f3993d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53764bbdcef0495baed9b6d7c9dc1302"
I20260809 04:39:17.351161 15767 raft_consensus.cc:3037] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:17.351197 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ed7d217204e608d9354c398d34629" candidate_uuid: "9c9ffce5989e449ca1395abd66f3993d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1659769e6e449c186ef20ddd9ee75ac"
I20260809 04:39:17.351310 15963 raft_consensus.cc:3037] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:17.352217 15767 raft_consensus.cc:2454] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c9ffce5989e449ca1395abd66f3993d in term 1.
I20260809 04:39:17.352427 15963 raft_consensus.cc:2454] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c9ffce5989e449ca1395abd66f3993d in term 1.
I20260809 04:39:17.352536 16053 leader_election.cc:302] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53764bbdcef0495baed9b6d7c9dc1302, 9c9ffce5989e449ca1395abd66f3993d; no voters:
I20260809 04:39:17.352733 16272 raft_consensus.cc:2781] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:17.352963 16272 raft_consensus.cc:686] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 1 LEADER]: Becoming Leader. State: Replica: 9c9ffce5989e449ca1395abd66f3993d, State: Running, Role: LEADER
I20260809 04:39:17.353355 16272 consensus_queue.cc:237] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } }
I20260809 04:39:17.356586 15556 catalog_manager.cc:5495] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c9ffce5989e449ca1395abd66f3993d (127.19.92.3). New cstate: current_term: 1 leader_uuid: "9c9ffce5989e449ca1395abd66f3993d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c9ffce5989e449ca1395abd66f3993d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 39871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 } health_report { overall_health: UNKNOWN } } }
W20260809 04:39:17.374645 16237 tablet.cc:2369] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:17.404049 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.400778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.19.92.3:39871)
0809 04:39:17.400900 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:17.400903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:17.400916 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:39:17.400989 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:17.400992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:17.401048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:17.401135 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:17.401399 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.401922 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.402720 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:17.402935 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:17.403015 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:17.403262 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:17.403691 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:17.403798 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:17.403871 (+ 73us) server_negotiation.cc:299] Negotiation successful
0809 04:39:17.403906 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 04:39:17.407194 16026 tablet.cc:2369] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:17.410547 15963 raft_consensus.cc:1261] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 1 FOLLOWER]: Refusing update from remote peer 9c9ffce5989e449ca1395abd66f3993d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:17.410547 15767 raft_consensus.cc:1261] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [term 1 FOLLOWER]: Refusing update from remote peer 9c9ffce5989e449ca1395abd66f3993d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:17.411294 16272 consensus_queue.cc:1035] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53764bbdcef0495baed9b6d7c9dc1302" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 46599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:17.411504 16279 consensus_queue.cc:1035] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1659769e6e449c186ef20ddd9ee75ac" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 44475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:17.421631 16286 mvcc.cc:205] Tried to move back new op lower bound from 7316481463945441280 to 7316481463722172416. Current Snapshot: MvccSnapshot[applied={T|T < 7316481463945441280}]
I20260809 04:39:17.428061 15558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:17.428225 15558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:17.430815 15558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05006447f8fb460592d83d474a102b36: Sending DeleteTablet for 3 replicas of tablet b88ed7d217204e608d9354c398d34629
I20260809 04:39:17.431684 15932 tablet_service.cc:1514] Processing DeleteTablet for tablet b88ed7d217204e608d9354c398d34629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:17 UTC) from {username='slave'} at 127.0.0.1:59100
I20260809 04:39:17.431866 16133 tablet_service.cc:1514] Processing DeleteTablet for tablet b88ed7d217204e608d9354c398d34629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:17 UTC) from {username='slave'} at 127.0.0.1:38832
I20260809 04:39:17.432027 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet b88ed7d217204e608d9354c398d34629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:17 UTC) from {username='slave'} at 127.0.0.1:41836
I20260809 04:39:17.432544 16300 tablet_replica.cc:330] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302: stopping tablet replica
I20260809 04:39:17.432919 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 15644
I20260809 04:39:17.432978 16300 raft_consensus.cc:2229] T b88ed7d217204e608d9354c398d34629 P 53764bbdcef0495baed9b6d7c9dc1302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:17.437434 16296 tablet_replica.cc:330] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: stopping tablet replica
I20260809 04:39:17.437934 16296 raft_consensus.cc:2229] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:17.438366 16296 raft_consensus.cc:2258] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:17.441334 16296 ts_tablet_manager.cc:1905] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:17.444496 16296 ts_tablet_manager.cc:1918] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:17.444648 16296 log.cc:1196] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/b88ed7d217204e608d9354c398d34629
I20260809 04:39:17.445003 16296 ts_tablet_manager.cc:1939] T b88ed7d217204e608d9354c398d34629 P e1659769e6e449c186ef20ddd9ee75ac: Deleting consensus metadata
I20260809 04:39:17.445178 16297 tablet_replica.cc:330] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: stopping tablet replica
I20260809 04:39:17.445659 16297 raft_consensus.cc:2229] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:39:17.446169 16297 raft_consensus.cc:2258] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:17.446810 15541 catalog_manager.cc:4841] TS e1659769e6e449c186ef20ddd9ee75ac (127.19.92.2:44475): tablet b88ed7d217204e608d9354c398d34629 (table test-table [id=7a45be60f159438f947928970e1d2601]) successfully deleted
W20260809 04:39:17.447520 15542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:46599 (error 108)
I20260809 04:39:17.447777 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 15835
I20260809 04:39:17.452068 16297 ts_tablet_manager.cc:1905] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:17.461264 16297 ts_tablet_manager.cc:1918] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:17.461472 16297 log.cc:1196] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/b88ed7d217204e608d9354c398d34629
I20260809 04:39:17.461808 16297 ts_tablet_manager.cc:1939] T b88ed7d217204e608d9354c398d34629 P 9c9ffce5989e449ca1395abd66f3993d: Deleting consensus metadata
I20260809 04:39:17.463043 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:17.462718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:46599 (local address 127.0.0.1:41840)
0809 04:39:17.462850 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:17.462944 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 04:39:17.463243 15542 catalog_manager.cc:4584] TS 53764bbdcef0495baed9b6d7c9dc1302 (127.19.92.1:46599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b88ed7d217204e608d9354c398d34629: Network error: Client connection negotiation failed: client connection to 127.19.92.1:46599: connect: Connection refused (error 111)
I20260809 04:39:17.463908 15544 catalog_manager.cc:4841] TS 9c9ffce5989e449ca1395abd66f3993d (127.19.92.3:39871): tablet b88ed7d217204e608d9354c398d34629 (table test-table [id=7a45be60f159438f947928970e1d2601]) successfully deleted
I20260809 04:39:17.469237 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 16033
I20260809 04:39:17.480214 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 15521
2026-08-09T04:39:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1222 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:17 dist-test-slave-rmhg krb5kdc[16304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:17 dist-test-slave-rmhg krb5kdc[16304](info): set up 2 sockets
Aug 09 04:39:17 dist-test-slave-rmhg krb5kdc[16304](info): commencing operation
krb5kdc: starting...
W20260809 04:39:18.614274 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:18 dist-test-slave-rmhg krb5kdc[16304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:18.741344 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:43231
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:43231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:18.856758 16328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:18.857076 16328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:18.857118 16328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:18.861133 16328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:18.861215 16328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:18.861233 16328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:18.861249 16328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:18.861267 16328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:18.866585 16328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:43231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:43231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:18.867648 16328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:18.868724 16328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:18.877072 16334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:18.877082 16340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:18.877120 16328 server_base.cc:1034] running on GCE node
W20260809 04:39:18.877082 16336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:18.877887 16328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:18.878952 16328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:18.880121 16328 hybrid_clock.cc:648] HybridClock initialized: now 1786250358880102 us; error 30 us; skew 500 ppm
Aug 09 04:39:18 dist-test-slave-rmhg krb5kdc[16304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250358, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:18.882856 16328 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:18.883765 16328 webserver.cc:460] Webserver started at http://127.19.92.62:44271/ using document root <none> and password file <none>
I20260809 04:39:18.884339 16328 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:18.884423 16328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:18.884639 16328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:18.886802 16328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "298c7fdcfa3448618d59bcd6633cb412"
format_stamp: "Formatted at 2026-08-09 04:39:18 on dist-test-slave-rmhg"
server_key: "cfc4e67c39c62864392ad8986768d77d"
server_key_iv: "62d3792094bea0b9ff836196d3cc28d1"
server_key_version: "encryptionkey@0"
I20260809 04:39:18.887364 16328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "298c7fdcfa3448618d59bcd6633cb412"
format_stamp: "Formatted at 2026-08-09 04:39:18 on dist-test-slave-rmhg"
server_key: "cfc4e67c39c62864392ad8986768d77d"
server_key_iv: "62d3792094bea0b9ff836196d3cc28d1"
server_key_version: "encryptionkey@0"
I20260809 04:39:18.891242 16328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:39:18.893637 16349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:18.894871 16328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:18.895004 16328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "298c7fdcfa3448618d59bcd6633cb412"
format_stamp: "Formatted at 2026-08-09 04:39:18 on dist-test-slave-rmhg"
server_key: "cfc4e67c39c62864392ad8986768d77d"
server_key_iv: "62d3792094bea0b9ff836196d3cc28d1"
server_key_version: "encryptionkey@0"
I20260809 04:39:18.895098 16328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:18.911136 16328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:18.914163 16328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:18.914350 16328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:18.921327 16328 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:43231
I20260809 04:39:18.921377 16430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:43231 every 8 connection(s)
I20260809 04:39:18.922322 16328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:18.925329 16431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:18.928717 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 16328
I20260809 04:39:18.928866 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:18.929131 19824 external_mini_cluster.cc:1428] Setting key e5eecc5613ec024e1300f2b24d42fd57
I20260809 04:39:18.931743 16431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412: Bootstrap starting.
I20260809 04:39:18.934352 16431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:18.935289 16431 log.cc:824] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:18.937773 16431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412: No bootstrap required, opened a new log
Aug 09 04:39:18 dist-test-slave-rmhg krb5kdc[16304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:18.941124 16431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298c7fdcfa3448618d59bcd6633cb412" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43231 } }
I20260809 04:39:18.941403 16431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:18.941466 16431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298c7fdcfa3448618d59bcd6633cb412, State: Initialized, Role: FOLLOWER
I20260809 04:39:18.941948 16431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298c7fdcfa3448618d59bcd6633cb412" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43231 } }
I20260809 04:39:18.942090 16431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:18.942181 16431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:18.942296 16431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:18.943526 16431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298c7fdcfa3448618d59bcd6633cb412" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43231 } }
I20260809 04:39:18.943904 16431 leader_election.cc:302] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298c7fdcfa3448618d59bcd6633cb412; no voters:
I20260809 04:39:18.944424 16431 leader_election.cc:288] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:18.944515 16437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:18.944808 16437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [term 1 LEADER]: Becoming Leader. State: Replica: 298c7fdcfa3448618d59bcd6633cb412, State: Running, Role: LEADER
I20260809 04:39:18.944849 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:18.930637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.19.92.62:43231)
0809 04:39:18.931072 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:18.931089 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:18.931127 (+ 38us) server_negotiation.cc:407] Connection header received
0809 04:39:18.931760 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:18.931778 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:18.932161 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:18.932448 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:18.933397 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:18.934429 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:18.935585 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:18.936067 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:18.936184 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:18.938903 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:18.938908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:18.938923 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:18.938952 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:18.941295 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:18.941774 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:18.941779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:18.941784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:18.941900 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:18.942171 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:18.942174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:18.942176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:18.942746 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:18.942962 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:18.943188 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 04:39:18.943535 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20260809 04:39:18.945551 16437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298c7fdcfa3448618d59bcd6633cb412" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43231 } }
I20260809 04:39:18.945942 16431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:18.947268 16440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298c7fdcfa3448618d59bcd6633cb412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298c7fdcfa3448618d59bcd6633cb412" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43231 } } }
I20260809 04:39:18.947393 16440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:18.947793 16439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298c7fdcfa3448618d59bcd6633cb412. Latest consensus state: current_term: 1 leader_uuid: "298c7fdcfa3448618d59bcd6633cb412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298c7fdcfa3448618d59bcd6633cb412" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 43231 } } }
I20260809 04:39:18.947889 16439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:18.948102 16453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:18.950809 16453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:18.956343 16453 catalog_manager.cc:1347] Generated new cluster ID: 617dd2729a74403c857febac07f28f87
I20260809 04:39:18.956408 16453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:18.971645 16453 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:18.972631 16453 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:18.983844 16453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412: Generated new TSK 0
I20260809 04:39:18.984598 16453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.92.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
Entry for principal kudu/127.19.92.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab.
I20260809 04:39:19.016358 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43231
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:19.132630 16468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:19.132974 16468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:19.133025 16468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:19.137197 16468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:19.137275 16468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:19.137349 16468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:39:19.142771 16468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.1.keytab
--principal=kudu/127.19.92.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:19.143888 16468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:19.144981 16468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:19.154527 16474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:19.154592 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:19.154580 16468 server_base.cc:1034] running on GCE node
W20260809 04:39:19.154680 16475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:19.155153 16468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:19.155751 16468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:19.156903 16468 hybrid_clock.cc:648] HybridClock initialized: now 1786250359156871 us; error 46 us; skew 500 ppm
Aug 09 04:39:19 dist-test-slave-rmhg krb5kdc[16304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:19.159555 16468 init.cc:377] Logged in from keytab as kudu/127.19.92.1@KRBTEST.COM (short username kudu)
I20260809 04:39:19.160416 16468 webserver.cc:460] Webserver started at http://127.19.92.1:38145/ using document root <none> and password file <none>
I20260809 04:39:19.160954 16468 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:19.161015 16468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:19.161267 16468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:19.163395 16468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "979c0f7209211b50e474792106792de1"
server_key_iv: "b52c02d04a55e83bc3259678be4d9251"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.163973 16468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "979c0f7209211b50e474792106792de1"
server_key_iv: "b52c02d04a55e83bc3259678be4d9251"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.168218 16468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:39:19.170749 16490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.171985 16468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 04:39:19.172106 16468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "979c0f7209211b50e474792106792de1"
server_key_iv: "b52c02d04a55e83bc3259678be4d9251"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.172216 16468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:19.188284 16468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:19.191689 16468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:19.191879 16468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:19.192477 16468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:19.193385 16468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:19.193436 16468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.193490 16468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:19.193519 16468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.202255 16468 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:40423
I20260809 04:39:19.202284 16661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:40423 every 8 connection(s)
I20260809 04:39:19.203176 16468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:39:19.204161 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 16468
I20260809 04:39:19.204288 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:39:19.204588 19824 external_mini_cluster.cc:1428] Setting key bdb62558230b317ace5e530b2c5307cb
Aug 09 04:39:19 dist-test-slave-rmhg krb5kdc[16304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.1@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.2@KRBTEST.COM; defaulting to no policy
I20260809 04:39:19.217890 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.205177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:47055 (local address 127.19.92.62:43231)
0809 04:39:19.205337 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.205342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.206392 (+ 1050us) server_negotiation.cc:407] Connection header received
0809 04:39:19.208023 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.208027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.208078 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.208156 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:19.209499 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.210241 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.211393 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.211738 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.211845 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.214444 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:19.214449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:19.214452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:19.214484 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:19.215949 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:19.216450 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:19.216454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:19.216455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:19.216534 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:19.216853 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:19.216854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:19.216855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:19.217054 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:19.217123 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.217635 (+ 512us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.217757 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.2@KRBTEST.COM" created.
I20260809 04:39:19.218842 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.205674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43231 (local address 127.19.92.1:47055)
0809 04:39:19.206219 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:19.206283 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:19.207771 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:19.208381 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:19.208392 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:19.208800 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:19.209355 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.209372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.210347 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.210349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.211254 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.211265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.211873 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.211876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.212027 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.212623 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:19.212639 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:19.214263 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:19.216078 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:19.216084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:19.216097 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:19.216360 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:19.216615 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:19.216618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:19.216620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:19.216780 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:19.217124 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:19.217129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:19.217419 (+ 290us) client_negotiation.cc:769] Sending connection context
0809 04:39:19.217576 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 04:39:19.217873 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":144,"threads_started":1}
I20260809 04:39:19.220178 16662 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43231
I20260809 04:39:19.220409 16662 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:19.221036 16662 heartbeater.cc:511] Master 127.19.92.62:43231 requested a full tablet report, sending...
I20260809 04:39:19.222600 16376 ts_manager.cc:194] Registered new tserver with Master: e0cd0e5f94fc41389bc02f08d2a3e2e1 (127.19.92.1:40423)
I20260809 04:39:19.223794 16376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.1@KRBTEST.COM'} at 127.19.92.1:47055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
Entry for principal kudu/127.19.92.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab.
I20260809 04:39:19.236145 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43231
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:19.352535 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:19.352840 16668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:19.352891 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:19.356901 16668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:19.356961 16668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:19.357016 16668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:39:19.362361 16668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.2.keytab
--principal=kudu/127.19.92.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:19.363624 16668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:19.364732 16668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:19.374938 16683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:19.374986 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:19.375187 16668 server_base.cc:1034] running on GCE node
W20260809 04:39:19.374928 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:19.375716 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:19.376425 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:19.377580 16668 hybrid_clock.cc:648] HybridClock initialized: now 1786250359377553 us; error 45 us; skew 500 ppm
Aug 09 04:39:19 dist-test-slave-rmhg krb5kdc[16304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:19.380213 16668 init.cc:377] Logged in from keytab as kudu/127.19.92.2@KRBTEST.COM (short username kudu)
I20260809 04:39:19.381086 16668 webserver.cc:460] Webserver started at http://127.19.92.2:37809/ using document root <none> and password file <none>
I20260809 04:39:19.381649 16668 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:19.381711 16668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:19.381891 16668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:19.383906 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "60146fdaa7fe46e3ba115bd51c1024cb"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "e3aeddb614e6889f1155e45a0601a75c"
server_key_iv: "18dd9e352be021628b4efc2823e9b414"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.384433 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "60146fdaa7fe46e3ba115bd51c1024cb"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "e3aeddb614e6889f1155e45a0601a75c"
server_key_iv: "18dd9e352be021628b4efc2823e9b414"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.388221 16668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:39:19.390590 16692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.391772 16668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:39:19.391878 16668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "60146fdaa7fe46e3ba115bd51c1024cb"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "e3aeddb614e6889f1155e45a0601a75c"
server_key_iv: "18dd9e352be021628b4efc2823e9b414"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.391961 16668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:19.407668 16668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:19.410811 16668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:19.410981 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:19.411573 16668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:19.412420 16668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:19.412472 16668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.412523 16668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:19.412542 16668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.421175 16668 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:40763
I20260809 04:39:19.421224 16864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:40763 every 8 connection(s)
I20260809 04:39:19.422137 16668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:39:19.424036 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 16668
I20260809 04:39:19.424177 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:39:19.424472 19824 external_mini_cluster.cc:1428] Setting key c984f79c3ecca2b53b7fce702c2b8d76
Aug 09 04:39:19 dist-test-slave-rmhg krb5kdc[16304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.2@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.3@KRBTEST.COM; defaulting to no policy
I20260809 04:39:19.436182 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.423990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:52959 (local address 127.19.92.62:43231)
0809 04:39:19.424144 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.424149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.425086 (+ 937us) server_negotiation.cc:407] Connection header received
0809 04:39:19.426424 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.426427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.426478 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.426563 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:19.427783 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.428377 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.429836 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.430080 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.430176 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.432602 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:19.432606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:19.432608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:19.432632 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:19.434103 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:19.434641 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:19.434645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:19.434647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:19.434721 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:19.435030 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:19.435033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:19.435035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:19.435223 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:19.435310 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.435859 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.436005 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.3@KRBTEST.COM" created.
I20260809 04:39:19.437125 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.424439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43231 (local address 127.19.92.2:52959)
0809 04:39:19.424926 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:19.424979 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:19.426203 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:19.426718 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:19.426725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:19.427091 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:19.427632 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.427646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.428499 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.428503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.429719 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.429729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.430225 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.430229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.430395 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.430976 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:19.430994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:19.432440 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:19.434214 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:19.434220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:19.434236 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:19.434552 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:19.434804 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:19.434807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:19.434809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:19.434945 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:19.435325 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:19.435331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:19.435627 (+ 296us) client_negotiation.cc:769] Sending connection context
0809 04:39:19.435799 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 04:39:19.436165 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":133,"threads_started":1}
I20260809 04:39:19.438373 16866 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43231
I20260809 04:39:19.438651 16866 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:19.439236 16866 heartbeater.cc:511] Master 127.19.92.62:43231 requested a full tablet report, sending...
I20260809 04:39:19.440544 16376 ts_manager.cc:194] Registered new tserver with Master: 60146fdaa7fe46e3ba115bd51c1024cb (127.19.92.2:40763)
I20260809 04:39:19.441115 16376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.2@KRBTEST.COM'} at 127.19.92.2:52959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
Entry for principal kudu/127.19.92.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab.
I20260809 04:39:19.454412 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43231
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:19.572302 16875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:19.572600 16875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:19.572664 16875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:19.577113 16875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:19.577224 16875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:19.577314 16875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:39:19.583052 16875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.3.keytab
--principal=kudu/127.19.92.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:43231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:19.584210 16875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:19.585362 16875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:19.595278 16883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:19.595299 16887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:19.595261 16882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:19.595625 16875 server_base.cc:1034] running on GCE node
I20260809 04:39:19.596064 16875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:19.596725 16875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:19.597887 16875 hybrid_clock.cc:648] HybridClock initialized: now 1786250359597859 us; error 44 us; skew 500 ppm
Aug 09 04:39:19 dist-test-slave-rmhg krb5kdc[16304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:19.600793 16875 init.cc:377] Logged in from keytab as kudu/127.19.92.3@KRBTEST.COM (short username kudu)
I20260809 04:39:19.601830 16875 webserver.cc:460] Webserver started at http://127.19.92.3:41443/ using document root <none> and password file <none>
I20260809 04:39:19.602399 16875 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:19.602469 16875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:19.602705 16875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:19.605012 16875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "11d7ffd980824385960f1f55999f3526"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "bbad5e10a692ef2ada02d115ec8e386d"
server_key_iv: "ff06546e55b4afa47b29307a283b2c53"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.605731 16875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "11d7ffd980824385960f1f55999f3526"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "bbad5e10a692ef2ada02d115ec8e386d"
server_key_iv: "ff06546e55b4afa47b29307a283b2c53"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.609979 16875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:39:19.612658 16896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.613991 16875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:39:19.614116 16875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "11d7ffd980824385960f1f55999f3526"
format_stamp: "Formatted at 2026-08-09 04:39:19 on dist-test-slave-rmhg"
server_key: "bbad5e10a692ef2ada02d115ec8e386d"
server_key_iv: "ff06546e55b4afa47b29307a283b2c53"
server_key_version: "encryptionkey@0"
I20260809 04:39:19.614228 16875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:19.643124 16875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:19.646894 16875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:19.647101 16875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:19.647893 16875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:19.648901 16875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:19.648969 16875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.649027 16875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:19.649055 16875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:19.658179 16875 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:41311
I20260809 04:39:19.658250 17062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:41311 every 8 connection(s)
I20260809 04:39:19.659171 16875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:39:19.662716 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 16875
I20260809 04:39:19.662858 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:39:19.663108 19824 external_mini_cluster.cc:1428] Setting key 9187743a8cb8c500f028fb3fc6a41247
Aug 09 04:39:19 dist-test-slave-rmhg krb5kdc[16304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250359, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.3@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:19.673390 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.661119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:43413 (local address 127.19.92.62:43231)
0809 04:39:19.661281 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.661285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.662193 (+ 908us) server_negotiation.cc:407] Connection header received
0809 04:39:19.663677 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.663680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.663733 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.663820 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:19.665422 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.665967 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.667206 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.667420 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.667508 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.669940 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:19.669943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:19.669945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:19.669972 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:19.671298 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:19.671824 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:19.671827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:19.671828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:19.671915 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:19.672255 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:19.672257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:19.672258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:19.672447 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:19.672534 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.673065 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.673224 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:39:19.674220 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.661537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:43231 (local address 127.19.92.3:43413)
0809 04:39:19.662039 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:19.662085 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:19.663354 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:19.664044 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:19.664054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:19.664576 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:19.665287 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.665307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.666151 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.666154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.667070 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.667080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.667558 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.667562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.667757 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.668287 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:19.668304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:19.669777 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:19.671431 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:19.671438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:19.671454 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:19.671718 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:19.671998 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:19.672001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:19.672003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:19.672157 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:19.672549 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:19.672556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:19.672809 (+ 253us) client_negotiation.cc:769] Sending connection context
0809 04:39:19.672979 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 04:39:19.673375 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
I20260809 04:39:19.675343 17064 heartbeater.cc:347] Connected to a master server at 127.19.92.62:43231
I20260809 04:39:19.675606 17064 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:19.676210 17064 heartbeater.cc:511] Master 127.19.92.62:43231 requested a full tablet report, sending...
I20260809 04:39:19.677326 16376 ts_manager.cc:194] Registered new tserver with Master: 11d7ffd980824385960f1f55999f3526 (127.19.92.3:41311)
I20260809 04:39:19.677881 16376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.92.3@KRBTEST.COM'} at 127.19.92.3:43413
I20260809 04:39:19.689375 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:39:19.698632 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.691423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.19.92.62:43231)
0809 04:39:19.691575 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.691580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.691714 (+ 134us) server_negotiation.cc:407] Connection header received
0809 04:39:19.691822 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.691824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.691872 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.691940 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:19.692638 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.693228 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.694025 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.694244 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.694319 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.695680 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:19.695683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:19.695685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:19.695711 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:19.696901 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:19.697364 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:19.697367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:19.697368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:19.697448 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:19.697702 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:19.697706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:19.697708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:19.697943 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:19.698063 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.698354 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.698477 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:39:19.701563 16376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:39:19.704553 16376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:39:19.719529 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.713310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:41311 (local address 127.0.0.1:49370)
0809 04:39:19.714024 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:19.714038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:19.714116 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:19.715263 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:19.715266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:19.715285 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:19.715376 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.715382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.716833 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.716836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.717913 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.717923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.718923 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.718927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.719041 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.719081 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 04:39:19.719190 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 04:39:19.719348 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":40,"thread_start_us":90,"threads_started":1}
I20260809 04:39:19.719914 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.714233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.19.92.2:40763)
0809 04:39:19.714588 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.714600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.714793 (+ 193us) server_negotiation.cc:407] Connection header received
0809 04:39:19.714875 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.714880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.715066 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.715187 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:19.715401 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.716134 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.718087 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.718974 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.719179 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.719347 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.719632 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.719745 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260809 04:39:19.719575 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.713035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:40423 (local address 127.0.0.1:32944)
0809 04:39:19.713447 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:19.713500 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:19.713632 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:19.714635 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:19.714641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:19.714666 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:19.714831 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.714838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.716225 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.716228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.717938 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.717949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.719176 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.719179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.719297 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.719321 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:39:19.719374 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 04:39:19.719439 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":129,"spinlock_wait_cycles":1920,"thread_start_us":117,"threads_started":1}
I20260809 04:39:19.720151 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.713196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.19.92.1:40423)
0809 04:39:19.713644 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.713663 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.713689 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:39:19.713766 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.713771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.714463 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.714635 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:19.714939 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.716084 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.718087 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.718974 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.719692 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.719858 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.719914 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.720018 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":124,"threads_started":1}
I20260809 04:39:19.719758 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.713807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:40763 (local address 127.0.0.1:45394)
0809 04:39:19.714602 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:19.714614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:19.714707 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:19.715183 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:19.715185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:19.715203 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:19.715300 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.715307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.716253 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.716256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.717941 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.717949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.719171 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.719175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.719473 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.719498 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:39:19.719575 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:39:19.719643 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":240,"spinlock_wait_cycles":4864,"thread_start_us":96,"threads_started":1}
I20260809 04:39:19.720890 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.713887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.19.92.3:41311)
0809 04:39:19.714800 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.714812 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.714830 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:39:19.714892 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.714897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.715106 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.715519 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:19.715665 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.716709 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.718032 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.718807 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.718909 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.719091 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.719901 (+ 810us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.720694 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":145,"threads_started":1}
I20260809 04:39:19.723323 16776 tablet_service.cc:1467] Processing CreateTablet for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b (DEFAULT_TABLE table=test-table [id=0afffb7e87bc4ee8b45259942d44d0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:19.723444 16563 tablet_service.cc:1467] Processing CreateTablet for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b (DEFAULT_TABLE table=test-table [id=0afffb7e87bc4ee8b45259942d44d0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:19.723448 16968 tablet_service.cc:1467] Processing CreateTablet for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b (DEFAULT_TABLE table=test-table [id=0afffb7e87bc4ee8b45259942d44d0cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:19.724553 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:19.724553 16776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:19.724565 16968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:19.731280 17090 tablet_bootstrap.cc:492] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Bootstrap starting.
I20260809 04:39:19.732506 17093 tablet_bootstrap.cc:492] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Bootstrap starting.
I20260809 04:39:19.733664 17091 tablet_bootstrap.cc:492] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1: Bootstrap starting.
I20260809 04:39:19.734102 17090 tablet_bootstrap.cc:654] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:19.734635 17093 tablet_bootstrap.cc:654] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:19.735148 17090 log.cc:824] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:19.735831 17093 log.cc:824] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:19.735877 17091 tablet_bootstrap.cc:654] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:19.736670 17091 log.cc:824] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:19.737401 17090 tablet_bootstrap.cc:492] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: No bootstrap required, opened a new log
I20260809 04:39:19.737612 17090 ts_tablet_manager.cc:1397] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 04:39:19.737932 17093 tablet_bootstrap.cc:492] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: No bootstrap required, opened a new log
I20260809 04:39:19.738121 17093 ts_tablet_manager.cc:1397] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:39:19.738489 17091 tablet_bootstrap.cc:492] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1: No bootstrap required, opened a new log
I20260809 04:39:19.738667 17091 ts_tablet_manager.cc:1397] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 04:39:19.741638 17090 raft_consensus.cc:348] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.742022 17090 raft_consensus.cc:374] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:19.741856 17093 raft_consensus.cc:348] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.741882 17091 raft_consensus.cc:348] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.742112 17090 raft_consensus.cc:729] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60146fdaa7fe46e3ba115bd51c1024cb, State: Initialized, Role: FOLLOWER
I20260809 04:39:19.742129 17091 raft_consensus.cc:374] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:19.742143 17093 raft_consensus.cc:374] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:19.742177 17091 raft_consensus.cc:729] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cd0e5f94fc41389bc02f08d2a3e2e1, State: Initialized, Role: FOLLOWER
I20260809 04:39:19.742189 17093 raft_consensus.cc:729] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d7ffd980824385960f1f55999f3526, State: Initialized, Role: FOLLOWER
I20260809 04:39:19.742656 17091 consensus_queue.cc:260] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.742655 17090 consensus_queue.cc:260] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.742655 17093 consensus_queue.cc:260] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.743501 17064 heartbeater.cc:503] Master 127.19.92.62:43231 was elected leader, sending a full tablet report...
I20260809 04:39:19.743698 17093 ts_tablet_manager.cc:1428] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:39:19.743750 17091 ts_tablet_manager.cc:1428] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 04:39:19.743914 16866 heartbeater.cc:503] Master 127.19.92.62:43231 was elected leader, sending a full tablet report...
I20260809 04:39:19.744165 17090 ts_tablet_manager.cc:1428] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:39:19.745082 16662 heartbeater.cc:503] Master 127.19.92.62:43231 was elected leader, sending a full tablet report...
W20260809 04:39:19.910166 17065 tablet.cc:2369] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:39:19.923303 16867 tablet.cc:2369] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:39:19.954545 16663 tablet.cc:2369] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:19.970659 17103 raft_consensus.cc:482] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:19.970877 17103 raft_consensus.cc:504] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.972014 17103 leader_election.cc:288] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60146fdaa7fe46e3ba115bd51c1024cb (127.19.92.2:40763), 11d7ffd980824385960f1f55999f3526 (127.19.92.3:41311)
I20260809 04:39:19.976089 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.972297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:40763 (local address 127.19.92.1:32855)
0809 04:39:19.972653 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:19.972669 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:19.972791 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:19.973081 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:19.973084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:19.973099 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:19.973281 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.973288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.974049 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.974052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.975057 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.975067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.975673 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.975677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.975769 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.975793 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:39:19.975854 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 04:39:19.975910 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":79,"thread_start_us":131,"threads_started":1}
I20260809 04:39:19.976099 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.972423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:32855 (local address 127.19.92.2:40763)
0809 04:39:19.972560 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.972565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.972755 (+ 190us) server_negotiation.cc:407] Connection header received
0809 04:39:19.972931 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.972935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.972987 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.973046 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:19.973379 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.973941 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.975174 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.975555 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.975638 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.975709 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.975885 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.975972 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:39:19.976155 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.972434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:50223 (local address 127.19.92.3:41311)
0809 04:39:19.972679 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:19.972684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:19.972848 (+ 164us) server_negotiation.cc:407] Connection header received
0809 04:39:19.973016 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:19.973019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:19.973079 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:19.973172 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:19.973536 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.974042 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.975143 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.975568 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.975686 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.975752 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:19.975953 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 04:39:19.976050 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260809 04:39:19.976119 17105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:19.972297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:41311 (local address 127.19.92.1:50223)
0809 04:39:19.972759 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:19.972773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:19.972884 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:19.973201 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:19.973204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:19.973220 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:19.973365 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.973372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.974145 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.974148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.975032 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:19.975041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:19.975723 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:19.975727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:19.975829 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:19.975854 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:39:19.975929 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 04:39:19.975990 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":106,"thread_start_us":135,"threads_started":1}
I20260809 04:39:19.976951 16991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dce6ef103cf49ef9f9b7f42d4f2030b" candidate_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d7ffd980824385960f1f55999f3526" is_pre_election: true
I20260809 04:39:19.976948 16803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dce6ef103cf49ef9f9b7f42d4f2030b" candidate_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" is_pre_election: true
I20260809 04:39:19.977321 16991 raft_consensus.cc:2454] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cd0e5f94fc41389bc02f08d2a3e2e1 in term 0.
I20260809 04:39:19.977321 16803 raft_consensus.cc:2454] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cd0e5f94fc41389bc02f08d2a3e2e1 in term 0.
I20260809 04:39:19.977799 16494 leader_election.cc:302] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60146fdaa7fe46e3ba115bd51c1024cb, e0cd0e5f94fc41389bc02f08d2a3e2e1; no voters:
I20260809 04:39:19.978055 17103 raft_consensus.cc:2781] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:39:19.978134 17103 raft_consensus.cc:482] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:39:19.978163 17103 raft_consensus.cc:3037] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:19.979194 17103 raft_consensus.cc:504] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.979550 17103 leader_election.cc:288] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [CANDIDATE]: Term 1 election: Requested vote from peers 60146fdaa7fe46e3ba115bd51c1024cb (127.19.92.2:40763), 11d7ffd980824385960f1f55999f3526 (127.19.92.3:41311)
I20260809 04:39:19.979913 16803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dce6ef103cf49ef9f9b7f42d4f2030b" candidate_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60146fdaa7fe46e3ba115bd51c1024cb"
I20260809 04:39:19.979911 16991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dce6ef103cf49ef9f9b7f42d4f2030b" candidate_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d7ffd980824385960f1f55999f3526"
I20260809 04:39:19.980036 16803 raft_consensus.cc:3037] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:19.980042 16991 raft_consensus.cc:3037] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:19.981051 16803 raft_consensus.cc:2454] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cd0e5f94fc41389bc02f08d2a3e2e1 in term 1.
I20260809 04:39:19.981134 16991 raft_consensus.cc:2454] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cd0e5f94fc41389bc02f08d2a3e2e1 in term 1.
I20260809 04:39:19.981361 16494 leader_election.cc:302] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60146fdaa7fe46e3ba115bd51c1024cb, e0cd0e5f94fc41389bc02f08d2a3e2e1; no voters:
I20260809 04:39:19.981582 17103 raft_consensus.cc:2781] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:19.981786 17103 raft_consensus.cc:686] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [term 1 LEADER]: Becoming Leader. State: Replica: e0cd0e5f94fc41389bc02f08d2a3e2e1, State: Running, Role: LEADER
I20260809 04:39:19.982141 17103 consensus_queue.cc:237] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } }
I20260809 04:39:19.985327 16374 catalog_manager.cc:5495] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0cd0e5f94fc41389bc02f08d2a3e2e1 (127.19.92.1). New cstate: current_term: 1 leader_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cd0e5f94fc41389bc02f08d2a3e2e1" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 40423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 } health_report { overall_health: UNKNOWN } } }
I20260809 04:39:20.013968 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.010542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.19.92.1:40423)
0809 04:39:20.010688 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.010693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.010705 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:39:20.010785 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.010787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.010845 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.010935 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:20.011161 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.011681 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.012505 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.012706 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.012775 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.013106 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:20.013596 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:20.013685 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:20.013768 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 04:39:20.013821 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:39:20.020018 16803 raft_consensus.cc:1261] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 1 FOLLOWER]: Refusing update from remote peer e0cd0e5f94fc41389bc02f08d2a3e2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:20.020007 16991 raft_consensus.cc:1261] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 1 FOLLOWER]: Refusing update from remote peer e0cd0e5f94fc41389bc02f08d2a3e2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:20.020728 17103 consensus_queue.cc:1035] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d7ffd980824385960f1f55999f3526" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 41311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:20.020934 17108 consensus_queue.cc:1035] T 5dce6ef103cf49ef9f9b7f42d4f2030b P e0cd0e5f94fc41389bc02f08d2a3e2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60146fdaa7fe46e3ba115bd51c1024cb" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 40763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:20.029899 17115 mvcc.cc:205] Tried to move back new op lower bound from 7316481474634846208 to 7316481474489319424. Current Snapshot: MvccSnapshot[applied={T|T < 7316481474634846208}]
I20260809 04:39:20.037926 16371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:20.038080 16371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:20.040508 16371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 298c7fdcfa3448618d59bcd6633cb412: Sending DeleteTablet for 3 replicas of tablet 5dce6ef103cf49ef9f9b7f42d4f2030b
I20260809 04:39:20.041337 16776 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:20 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:45394
I20260809 04:39:20.041323 16563 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:20 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:32944
I20260809 04:39:20.041380 16968 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:20 UTC) from {username='kudu', principal='kudu/127.19.92.62@KRBTEST.COM'} at 127.0.0.1:49370
I20260809 04:39:20.042044 17125 tablet_replica.cc:330] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: stopping tablet replica
I20260809 04:39:20.042369 17125 raft_consensus.cc:2229] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:20.042492 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 16468
I20260809 04:39:20.042702 17125 raft_consensus.cc:2258] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:20.043040 17122 tablet_replica.cc:330] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: stopping tablet replica
I20260809 04:39:20.043303 17122 raft_consensus.cc:2229] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:20.043560 17122 raft_consensus.cc:2258] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:20.048724 17125 ts_tablet_manager.cc:1905] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:20.051888 17125 ts_tablet_manager.cc:1918] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:20.052121 17125 log.cc:1196] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/5dce6ef103cf49ef9f9b7f42d4f2030b
I20260809 04:39:20.052642 17125 ts_tablet_manager.cc:1939] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 11d7ffd980824385960f1f55999f3526: Deleting consensus metadata
I20260809 04:39:20.053989 17122 ts_tablet_manager.cc:1905] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:20.054289 16354 catalog_manager.cc:4841] TS 11d7ffd980824385960f1f55999f3526 (127.19.92.3:41311): tablet 5dce6ef103cf49ef9f9b7f42d4f2030b (table test-table [id=0afffb7e87bc4ee8b45259942d44d0cb]) successfully deleted
I20260809 04:39:20.056243 17122 ts_tablet_manager.cc:1918] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:20.056334 17122 log.cc:1196] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/5dce6ef103cf49ef9f9b7f42d4f2030b
I20260809 04:39:20.056599 17122 ts_tablet_manager.cc:1939] T 5dce6ef103cf49ef9f9b7f42d4f2030b P 60146fdaa7fe46e3ba115bd51c1024cb: Deleting consensus metadata
I20260809 04:39:20.057821 16352 catalog_manager.cc:4841] TS 60146fdaa7fe46e3ba115bd51c1024cb (127.19.92.2:40763): tablet 5dce6ef103cf49ef9f9b7f42d4f2030b (table test-table [id=0afffb7e87bc4ee8b45259942d44d0cb]) successfully deleted
I20260809 04:39:20.059177 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 16668
W20260809 04:39:20.059226 16352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:40423 (error 108)
I20260809 04:39:20.064343 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.063969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:40423 (local address 127.0.0.1:32970)
0809 04:39:20.064139 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:20.064239 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:40423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 04:39:20.064616 16352 catalog_manager.cc:4584] TS e0cd0e5f94fc41389bc02f08d2a3e2e1 (127.19.92.1:40423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5dce6ef103cf49ef9f9b7f42d4f2030b: Network error: Client connection negotiation failed: client connection to 127.19.92.1:40423: connect: Connection refused (error 111)
I20260809 04:39:20.071864 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 16875
I20260809 04:39:20.084079 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 16328
2026-08-09T04:39:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2606 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T04:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:20Z Disabled control of system clock
I20260809 04:39:20.124044 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41345
--webserver_interface=127.19.92.62
--webserver_port=0
--builtin_ntp_servers=127.19.92.20:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:39:20.239915 17132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:20.240221 17132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:20.240258 17132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:20.244316 17132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:20.244426 17132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:20.244442 17132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:20.244458 17132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:20.244473 17132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:20.249653 17132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:20.250842 17132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:20.252235 17132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:20.260771 17140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:20.260780 17137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:20.260780 17138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:20.260900 17132 server_base.cc:1034] running on GCE node
I20260809 04:39:20.261620 17132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:20.262703 17132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:20.263870 17132 hybrid_clock.cc:648] HybridClock initialized: now 1786250360263856 us; error 32 us; skew 500 ppm
I20260809 04:39:20.265769 17132 webserver.cc:460] Webserver started at http://127.19.92.62:36107/ using document root <none> and password file <none>
I20260809 04:39:20.266342 17132 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:20.266410 17132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:20.266666 17132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:20.268814 17132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "ad23c69dc1a34386ba1d64d6bb768b91"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "0c79fa611c0ea85c90808588fd625a81"
server_key_iv: "1ec8597d15972466b3158cd66bd56ec6"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.269447 17132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "ad23c69dc1a34386ba1d64d6bb768b91"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "0c79fa611c0ea85c90808588fd625a81"
server_key_iv: "1ec8597d15972466b3158cd66bd56ec6"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.273429 17132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:39:20.276110 17149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.277601 17132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:20.277734 17132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "ad23c69dc1a34386ba1d64d6bb768b91"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "0c79fa611c0ea85c90808588fd625a81"
server_key_iv: "1ec8597d15972466b3158cd66bd56ec6"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.277845 17132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:20.297636 17132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:20.298378 17132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:20.298573 17132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:20.305418 17132 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41345
I20260809 04:39:20.305433 17247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41345 every 8 connection(s)
I20260809 04:39:20.306391 17132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:20.309661 17249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:20.310833 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 17132
I20260809 04:39:20.310961 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:20.311249 19824 external_mini_cluster.cc:1428] Setting key 2653d04b36248276baaaafa2d74870ab
I20260809 04:39:20.315826 17249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91: Bootstrap starting.
I20260809 04:39:20.318192 17249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:20.318933 17249 log.cc:824] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:20.320564 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.312754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.19.92.62:41345)
0809 04:39:20.313509 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.313530 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.313568 (+ 38us) server_negotiation.cc:407] Connection header received
0809 04:39:20.314211 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.314228 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.314652 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.314963 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:20.315415 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.316437 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.317349 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.317821 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.317983 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.318794 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:20.318799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:20.318813 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:20.319229 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:20.319293 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:20.319322 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:20.319421 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:20.319584 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 04:39:20.319846 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
I20260809 04:39:20.321087 17249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91: No bootstrap required, opened a new log
I20260809 04:39:20.324848 17249 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad23c69dc1a34386ba1d64d6bb768b91" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41345 } }
I20260809 04:39:20.325168 17249 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:20.325248 17249 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad23c69dc1a34386ba1d64d6bb768b91, State: Initialized, Role: FOLLOWER
I20260809 04:39:20.325732 17249 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad23c69dc1a34386ba1d64d6bb768b91" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41345 } }
I20260809 04:39:20.325883 17249 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:20.325954 17249 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:20.326067 17249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:20.327214 17249 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad23c69dc1a34386ba1d64d6bb768b91" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41345 } }
I20260809 04:39:20.327593 17249 leader_election.cc:302] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad23c69dc1a34386ba1d64d6bb768b91; no voters:
I20260809 04:39:20.327934 17249 leader_election.cc:288] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:20.328063 17257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:20.328284 17257 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [term 1 LEADER]: Becoming Leader. State: Replica: ad23c69dc1a34386ba1d64d6bb768b91, State: Running, Role: LEADER
I20260809 04:39:20.328603 17257 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad23c69dc1a34386ba1d64d6bb768b91" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41345 } }
I20260809 04:39:20.329329 17249 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:20.330426 17259 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad23c69dc1a34386ba1d64d6bb768b91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad23c69dc1a34386ba1d64d6bb768b91" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41345 } } }
I20260809 04:39:20.330575 17259 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:20.330830 17260 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad23c69dc1a34386ba1d64d6bb768b91. Latest consensus state: current_term: 1 leader_uuid: "ad23c69dc1a34386ba1d64d6bb768b91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad23c69dc1a34386ba1d64d6bb768b91" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41345 } } }
I20260809 04:39:20.330925 17260 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:20.331092 17270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:20.334481 17270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:20.340430 17270 catalog_manager.cc:1347] Generated new cluster ID: 9dad4afe489048e9ab252018393cda2b
I20260809 04:39:20.340518 17270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:20.350548 17270 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:20.351565 17270 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:20.361280 17270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91: Generated new TSK 0
I20260809 04:39:20.362051 17270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:20.369846 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.1:0
--local_ip_for_outbound_sockets=127.19.92.1
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41345
--builtin_ntp_servers=127.19.92.20:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:39:20.490382 17285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:20.490697 17285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:20.490738 17285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:20.494997 17285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:20.495075 17285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:20.495138 17285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.1
I20260809 04:39:20.500108 17285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.92.1
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.1
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:20.501286 17285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:20.502717 17285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:20.512486 17303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:20.512518 17295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:20.512488 17298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:20.512542 17285 server_base.cc:1034] running on GCE node
I20260809 04:39:20.513144 17285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:20.513885 17285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:20.515053 17285 hybrid_clock.cc:648] HybridClock initialized: now 1786250360515031 us; error 34 us; skew 500 ppm
I20260809 04:39:20.516934 17285 webserver.cc:460] Webserver started at http://127.19.92.1:38903/ using document root <none> and password file <none>
I20260809 04:39:20.517590 17285 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:20.517668 17285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:20.517889 17285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:20.520017 17285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/data/instance:
uuid: "0833b66f20e5456ca51eaf66f705fe08"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "5755addd1a6eb0449f96452dd18f5619"
server_key_iv: "84ba2072e7920c04f19a2f180d308146"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.520591 17285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance:
uuid: "0833b66f20e5456ca51eaf66f705fe08"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "5755addd1a6eb0449f96452dd18f5619"
server_key_iv: "84ba2072e7920c04f19a2f180d308146"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.524721 17285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:39:20.527626 17314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.529054 17285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:39:20.529318 17285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/wal
uuid: "0833b66f20e5456ca51eaf66f705fe08"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "5755addd1a6eb0449f96452dd18f5619"
server_key_iv: "84ba2072e7920c04f19a2f180d308146"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.529502 17285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:20.549995 17285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:20.550922 17285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:20.551103 17285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:20.551697 17285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:20.552704 17285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:20.552752 17285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.552793 17285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:20.552812 17285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.561563 17285 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.1:43465
I20260809 04:39:20.561592 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.1:43465 every 8 connection(s)
I20260809 04:39:20.562574 17285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/data/info.pb
I20260809 04:39:20.568085 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 17285
I20260809 04:39:20.568226 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-0/wal/instance
I20260809 04:39:20.568518 19824 external_mini_cluster.cc:1428] Setting key 7d7f87f730449a6eb5bc6f07fba57c33
I20260809 04:39:20.570642 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.2:0
--local_ip_for_outbound_sockets=127.19.92.2
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41345
--builtin_ntp_servers=127.19.92.20:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:39:20.572934 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.564670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.1:49819 (local address 127.19.92.62:41345)
0809 04:39:20.564819 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.564823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.566295 (+ 1472us) server_negotiation.cc:407] Connection header received
0809 04:39:20.567763 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.567767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.567833 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.567913 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:20.568984 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.569668 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.571009 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.571260 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.571350 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.572465 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:20.572470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:20.572473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:20.572489 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:20.572514 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:20.572525 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:20.572591 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:20.572748 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 04:39:20.572805 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:39:20.574128 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.565246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41345 (local address 127.19.92.1:49819)
0809 04:39:20.566096 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:20.566144 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:20.567511 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:20.568127 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:20.568135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:20.568233 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:20.568828 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.568844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.569803 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.569805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:20.570882 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.570893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.571415 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.571419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:20.571584 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.572167 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:20.572169 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:20.572214 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:20.572284 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:20.572605 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:20.572614 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:20.572620 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:39:20.572752 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 04:39:20.573078 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":186,"threads_started":1}
I20260809 04:39:20.575361 17496 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41345
I20260809 04:39:20.575608 17496 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:20.576145 17496 heartbeater.cc:511] Master 127.19.92.62:41345 requested a full tablet report, sending...
I20260809 04:39:20.577920 17182 ts_manager.cc:194] Registered new tserver with Master: 0833b66f20e5456ca51eaf66f705fe08 (127.19.92.1:43465)
I20260809 04:39:20.579532 17182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.1:49819
W20260809 04:39:20.686514 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:20.686820 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:20.686857 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:20.691022 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:20.691092 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:20.691149 17501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.2
I20260809 04:39:20.695932 17501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.92.2
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.2
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:20.696995 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:20.698454 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:20.707713 17509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:20.707839 17501 server_base.cc:1034] running on GCE node
W20260809 04:39:20.707703 17508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:20.707703 17512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:20.708449 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:20.709223 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:20.710386 17501 hybrid_clock.cc:648] HybridClock initialized: now 1786250360710371 us; error 25 us; skew 500 ppm
I20260809 04:39:20.712153 17501 webserver.cc:460] Webserver started at http://127.19.92.2:43369/ using document root <none> and password file <none>
I20260809 04:39:20.712729 17501 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:20.712792 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:20.713013 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:20.715219 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/data/instance:
uuid: "6924239ada0e4feb8b38e633da5620fe"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "7ed10e92df4f187dc31ab5b695d6975f"
server_key_iv: "39cfc4088f08d220a5b78773294af1ff"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.715798 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance:
uuid: "6924239ada0e4feb8b38e633da5620fe"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "7ed10e92df4f187dc31ab5b695d6975f"
server_key_iv: "39cfc4088f08d220a5b78773294af1ff"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.719660 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 04:39:20.722249 17520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.723508 17501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:20.723614 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/wal
uuid: "6924239ada0e4feb8b38e633da5620fe"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "7ed10e92df4f187dc31ab5b695d6975f"
server_key_iv: "39cfc4088f08d220a5b78773294af1ff"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.723734 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:20.751871 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:20.752605 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:20.752782 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:20.753448 17501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:20.754438 17501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:20.754487 17501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.754527 17501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:20.754557 17501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.762876 17501 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.2:39675
I20260809 04:39:20.762905 17679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.2:39675 every 8 connection(s)
I20260809 04:39:20.763784 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/data/info.pb
I20260809 04:39:20.769235 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 17501
I20260809 04:39:20.769393 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/wal/instance
I20260809 04:39:20.769747 19824 external_mini_cluster.cc:1428] Setting key 54fb24b8f5653257e9309f9cbffcbd75
I20260809 04:39:20.772724 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.92.3:0
--local_ip_for_outbound_sockets=127.19.92.3
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41345
--builtin_ntp_servers=127.19.92.20:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:39:20.776542 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.765696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.2:51613 (local address 127.19.92.62:41345)
0809 04:39:20.765844 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.765848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.767215 (+ 1367us) server_negotiation.cc:407] Connection header received
0809 04:39:20.769202 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.769207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.769274 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.769365 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:20.770853 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.771690 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.773763 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.774186 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.774278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.775924 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:20.775928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:20.775940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:20.775959 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:20.775985 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:20.775998 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:20.776089 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:20.776316 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 04:39:20.776395 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:39:20.777760 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.766137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41345 (local address 127.19.92.2:51613)
0809 04:39:20.767028 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:20.767087 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:20.768874 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:20.769727 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:20.769737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:20.769945 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:20.770678 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.770697 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.771863 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.771867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:20.773590 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.773604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.774527 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.774532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:20.774783 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.775544 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:20.775546 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:20.775617 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:20.775713 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:20.776101 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:20.776111 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:20.776117 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:39:20.776293 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 04:39:20.776633 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":142,"threads_started":1}
I20260809 04:39:20.779436 17682 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41345
I20260809 04:39:20.779695 17682 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:20.780224 17682 heartbeater.cc:511] Master 127.19.92.62:41345 requested a full tablet report, sending...
I20260809 04:39:20.781327 17182 ts_manager.cc:194] Registered new tserver with Master: 6924239ada0e4feb8b38e633da5620fe (127.19.92.2:39675)
I20260809 04:39:20.781904 17182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.2:51613
W20260809 04:39:20.892036 17685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:20.892340 17685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:20.892396 17685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:20.896566 17685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:20.896638 17685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:20.896718 17685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.92.3
I20260809 04:39:20.901715 17685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.92.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.92.3
--webserver_port=0
--tserver_master_addrs=127.19.92.62:41345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.92.3
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:20.902899 17685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:20.904349 17685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:20.914052 17697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:20.914063 17693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:20.914165 17694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:20.914206 17685 server_base.cc:1034] running on GCE node
I20260809 04:39:20.914705 17685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:20.915341 17685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:20.916522 17685 hybrid_clock.cc:648] HybridClock initialized: now 1786250360916495 us; error 39 us; skew 500 ppm
I20260809 04:39:20.918421 17685 webserver.cc:460] Webserver started at http://127.19.92.3:42479/ using document root <none> and password file <none>
I20260809 04:39:20.919020 17685 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:20.919093 17685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:20.919306 17685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:20.921487 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/data/instance:
uuid: "e5684a95375846f1bb2a9feaa078c09d"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "f6e022e2e5da6eb87930d7633a64ae5f"
server_key_iv: "6836dc61f2ec6b8ab94c72efa0a91d6d"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.922062 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance:
uuid: "e5684a95375846f1bb2a9feaa078c09d"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "f6e022e2e5da6eb87930d7633a64ae5f"
server_key_iv: "6836dc61f2ec6b8ab94c72efa0a91d6d"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.926038 17685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:39:20.928860 17704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.930266 17685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:39:20.930399 17685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/wal
uuid: "e5684a95375846f1bb2a9feaa078c09d"
format_stamp: "Formatted at 2026-08-09 04:39:20 on dist-test-slave-rmhg"
server_key: "f6e022e2e5da6eb87930d7633a64ae5f"
server_key_iv: "6836dc61f2ec6b8ab94c72efa0a91d6d"
server_key_version: "encryptionkey@0"
I20260809 04:39:20.930536 17685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:20.949790 17685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:20.950574 17685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:20.950771 17685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:20.951442 17685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:39:20.952461 17685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:39:20.952512 17685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.952566 17685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:39:20.952594 17685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:20.961300 17685 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.3:44835
I20260809 04:39:20.961331 17874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.3:44835 every 8 connection(s)
I20260809 04:39:20.962260 17685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/data/info.pb
I20260809 04:39:20.962460 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 17685
I20260809 04:39:20.962584 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/wal/instance
I20260809 04:39:20.962930 19824 external_mini_cluster.cc:1428] Setting key dcca08c8cff04492531afd49104e8475
I20260809 04:39:20.971931 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.964180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:38989 (local address 127.19.92.62:41345)
0809 04:39:20.964318 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.964321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.965696 (+ 1375us) server_negotiation.cc:407] Connection header received
0809 04:39:20.967132 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.967135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.967188 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.967262 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:20.968245 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.968805 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.970117 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.970350 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.970433 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.971530 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:20.971533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:20.971535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:20.971550 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:20.971569 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:20.971577 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:20.971633 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:20.971749 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 04:39:20.971808 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:39:20.972807 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.964708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:41345 (local address 127.19.92.3:38989)
0809 04:39:20.965531 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:20.965588 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:20.966896 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:20.967435 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:20.967443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:20.967536 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:20.968104 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.968119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.968955 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.968958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:20.969991 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.970002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.970487 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.970490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:20.970647 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.971246 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:39:20.971247 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:20.971293 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:39:20.971367 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:20.971639 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:20.971644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:20.971648 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:39:20.971743 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 04:39:20.972000 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
I20260809 04:39:20.973986 17877 heartbeater.cc:347] Connected to a master server at 127.19.92.62:41345
I20260809 04:39:20.974224 17877 heartbeater.cc:464] Registering TS with master...
I20260809 04:39:20.974763 17877 heartbeater.cc:511] Master 127.19.92.62:41345 requested a full tablet report, sending...
I20260809 04:39:20.975816 17182 ts_manager.cc:194] Registered new tserver with Master: e5684a95375846f1bb2a9feaa078c09d (127.19.92.3:44835)
I20260809 04:39:20.976440 17182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.92.3:38989
I20260809 04:39:20.977035 19824 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:39:20.982714 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.979013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.19.92.62:41345)
0809 04:39:20.979140 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.979145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.979340 (+ 195us) server_negotiation.cc:407] Connection header received
0809 04:39:20.979457 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.979460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.979534 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.979600 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:20.979882 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.980391 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.981256 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.981509 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.981596 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:20.982322 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:20.982326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:20.982328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:20.982342 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:20.982362 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:20.982372 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:20.982431 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:20.982527 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 04:39:20.982581 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:39:20.985437 17183 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:39:20.988288 17183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:39:21.001852 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.996464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:43465 (local address 127.0.0.1:58908)
0809 04:39:20.996800 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:20.996856 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:20.996980 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:20.997810 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:20.997816 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:20.997842 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:20.997999 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.998006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.999260 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:20.999263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.000224 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:21.000235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.001327 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.001330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.001436 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.001477 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 04:39:21.001588 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 04:39:21.001703 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260809 04:39:21.001899 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.996642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.19.92.1:43465)
0809 04:39:20.997090 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.997103 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.997128 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:39:20.997240 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.997245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.997467 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.997630 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:20.998123 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.999160 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.000363 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.001213 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.001312 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.001536 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:21.001629 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 04:39:21.001728 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":121,"threads_started":1}
I20260809 04:39:21.002722 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.997484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:39675 (local address 127.0.0.1:44782)
0809 04:39:20.998062 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:20.998074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:20.998229 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:20.998700 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:20.998703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:20.998721 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:20.998821 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.998827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.000103 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.000106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.001187 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:21.001198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.002343 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.002347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.002447 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.002466 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 04:39:21.002532 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 04:39:21.002598 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":24,"thread_start_us":100,"threads_started":1}
I20260809 04:39:21.002817 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.997375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.19.92.2:39675)
0809 04:39:20.997713 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.997721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.998159 (+ 438us) server_negotiation.cc:407] Connection header received
0809 04:39:20.998358 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.998363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.998525 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.998669 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:20.998926 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.000004 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.001297 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.002074 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.002174 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.002352 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:21.002571 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 04:39:21.002665 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
I20260809 04:39:21.002884 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.996784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.3:44835 (local address 127.0.0.1:56526)
0809 04:39:20.997901 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:20.997914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:20.998002 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:20.998525 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:20.998528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:20.998544 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:20.998651 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:20.998657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.000065 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.000069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.001280 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:21.001290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.002514 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.002518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.002606 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.002629 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:39:21.002695 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 04:39:21.002759 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":34,"thread_start_us":103,"threads_started":1}
I20260809 04:39:21.003342 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:20.997305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.19.92.3:44835)
0809 04:39:20.997906 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:20.997917 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:20.998102 (+ 185us) server_negotiation.cc:407] Connection header received
0809 04:39:20.998185 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:20.998191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:20.998382 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:20.998542 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:20.998773 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:20.999931 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.001407 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.002170 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.002271 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.002449 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:21.003114 (+ 665us) server_negotiation.cc:299] Negotiation successful
0809 04:39:21.003214 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":99,"threads_started":1}
I20260809 04:39:21.005299 17385 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd8a4c972464baaadac2efa5478b815 (DEFAULT_TABLE table=test-table [id=0176b7453f464e6a873b1b013b1b9ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:21.006122 17775 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd8a4c972464baaadac2efa5478b815 (DEFAULT_TABLE table=test-table [id=0176b7453f464e6a873b1b013b1b9ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:21.006608 17385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd8a4c972464baaadac2efa5478b815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:21.007081 17775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd8a4c972464baaadac2efa5478b815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:21.007112 17582 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd8a4c972464baaadac2efa5478b815 (DEFAULT_TABLE table=test-table [id=0176b7453f464e6a873b1b013b1b9ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:39:21.008164 17582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd8a4c972464baaadac2efa5478b815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:21.013357 17909 tablet_bootstrap.cc:492] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Bootstrap starting.
I20260809 04:39:21.014811 17910 tablet_bootstrap.cc:492] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08: Bootstrap starting.
I20260809 04:39:21.015828 17909 tablet_bootstrap.cc:654] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:21.016396 17911 tablet_bootstrap.cc:492] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Bootstrap starting.
I20260809 04:39:21.016937 17910 tablet_bootstrap.cc:654] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:21.016968 17909 log.cc:824] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:21.017740 17910 log.cc:824] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:21.018399 17911 tablet_bootstrap.cc:654] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:21.019219 17909 tablet_bootstrap.cc:492] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: No bootstrap required, opened a new log
I20260809 04:39:21.019204 17911 log.cc:824] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:21.019430 17910 tablet_bootstrap.cc:492] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08: No bootstrap required, opened a new log
I20260809 04:39:21.019424 17909 ts_tablet_manager.cc:1397] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 04:39:21.019613 17910 ts_tablet_manager.cc:1397] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 04:39:21.021279 17911 tablet_bootstrap.cc:492] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: No bootstrap required, opened a new log
I20260809 04:39:21.021453 17911 ts_tablet_manager.cc:1397] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260809 04:39:21.022945 17909 raft_consensus.cc:348] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.023160 17909 raft_consensus.cc:374] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:21.023213 17909 raft_consensus.cc:729] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5684a95375846f1bb2a9feaa078c09d, State: Initialized, Role: FOLLOWER
I20260809 04:39:21.023631 17909 consensus_queue.cc:260] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.023739 17910 raft_consensus.cc:348] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.024008 17910 raft_consensus.cc:374] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:21.024077 17910 raft_consensus.cc:729] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0833b66f20e5456ca51eaf66f705fe08, State: Initialized, Role: FOLLOWER
I20260809 04:39:21.024360 17877 heartbeater.cc:503] Master 127.19.92.62:41345 was elected leader, sending a full tablet report...
I20260809 04:39:21.024609 17909 ts_tablet_manager.cc:1428] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 04:39:21.024541 17910 consensus_queue.cc:260] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.025491 17910 ts_tablet_manager.cc:1428] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260809 04:39:21.025518 17911 raft_consensus.cc:348] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.025751 17911 raft_consensus.cc:374] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:21.025810 17911 raft_consensus.cc:729] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6924239ada0e4feb8b38e633da5620fe, State: Initialized, Role: FOLLOWER
I20260809 04:39:21.025820 17496 heartbeater.cc:503] Master 127.19.92.62:41345 was elected leader, sending a full tablet report...
I20260809 04:39:21.026245 17911 consensus_queue.cc:260] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.027240 17682 heartbeater.cc:503] Master 127.19.92.62:41345 was elected leader, sending a full tablet report...
I20260809 04:39:21.027807 17911 ts_tablet_manager.cc:1428] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260809 04:39:21.063781 17497 tablet.cc:2369] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:39:21.213478 17880 tablet.cc:2369] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:39:21.264950 17683 tablet.cc:2369] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:21.458068 17916 raft_consensus.cc:482] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:39:21.458294 17916 raft_consensus.cc:504] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.459430 17916 leader_election.cc:288] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0833b66f20e5456ca51eaf66f705fe08 (127.19.92.1:43465), 6924239ada0e4feb8b38e633da5620fe (127.19.92.2:39675)
I20260809 04:39:21.462965 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:21.459798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:60159 (local address 127.19.92.1:43465)
0809 04:39:21.459928 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:21.459933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:21.459953 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:39:21.460012 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:21.460014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:21.460067 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:21.460150 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:21.460457 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.460970 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.462016 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.462447 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.462547 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.462609 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:21.462781 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 04:39:21.462847 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:39:21.462966 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:21.459680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:43465 (local address 127.19.92.3:60159)
0809 04:39:21.459814 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:21.459829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:21.459927 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:21.460179 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:21.460183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:21.460205 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:21.460328 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:21.460335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.461078 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.461081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.461875 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:21.461885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.462564 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.462567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.462667 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.462687 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 04:39:21.462752 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 04:39:21.462824 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 04:39:21.463318 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:21.460112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.92.3:50379 (local address 127.19.92.2:39675)
0809 04:39:21.460236 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:21.460240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:21.460367 (+ 127us) server_negotiation.cc:407] Connection header received
0809 04:39:21.460491 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:21.460494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:21.460536 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:21.460595 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:39:21.460867 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.461398 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.462385 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.462760 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.462850 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.462909 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:21.463136 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 04:39:21.463206 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:39:21.463671 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:21.459963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.2:39675 (local address 127.19.92.3:50379)
0809 04:39:21.460292 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:21.460302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:21.460387 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:21.460618 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:21.460621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:21.460637 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:39:21.460765 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:21.460770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.461503 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.461506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.462268 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:21.462276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.462875 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.462879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:21.462980 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.463010 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 04:39:21.463465 (+ 455us) client_negotiation.cc:240] Negotiation successful
0809 04:39:21.463541 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":88,"threads_started":1}
I20260809 04:39:21.463661 17420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd8a4c972464baaadac2efa5478b815" candidate_uuid: "e5684a95375846f1bb2a9feaa078c09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0833b66f20e5456ca51eaf66f705fe08" is_pre_election: true
I20260809 04:39:21.463985 17420 raft_consensus.cc:2454] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5684a95375846f1bb2a9feaa078c09d in term 0.
I20260809 04:39:21.464298 17612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd8a4c972464baaadac2efa5478b815" candidate_uuid: "e5684a95375846f1bb2a9feaa078c09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6924239ada0e4feb8b38e633da5620fe" is_pre_election: true
I20260809 04:39:21.464540 17707 leader_election.cc:302] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0833b66f20e5456ca51eaf66f705fe08, e5684a95375846f1bb2a9feaa078c09d; no voters:
I20260809 04:39:21.464604 17612 raft_consensus.cc:2454] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5684a95375846f1bb2a9feaa078c09d in term 0.
I20260809 04:39:21.464812 17916 raft_consensus.cc:2781] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:39:21.464926 17916 raft_consensus.cc:482] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:39:21.464988 17916 raft_consensus.cc:3037] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:21.466061 17916 raft_consensus.cc:504] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.466387 17916 leader_election.cc:288] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [CANDIDATE]: Term 1 election: Requested vote from peers 0833b66f20e5456ca51eaf66f705fe08 (127.19.92.1:43465), 6924239ada0e4feb8b38e633da5620fe (127.19.92.2:39675)
I20260809 04:39:21.466727 17420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd8a4c972464baaadac2efa5478b815" candidate_uuid: "e5684a95375846f1bb2a9feaa078c09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0833b66f20e5456ca51eaf66f705fe08"
I20260809 04:39:21.466787 17612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd8a4c972464baaadac2efa5478b815" candidate_uuid: "e5684a95375846f1bb2a9feaa078c09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6924239ada0e4feb8b38e633da5620fe"
I20260809 04:39:21.466856 17420 raft_consensus.cc:3037] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:21.466894 17612 raft_consensus.cc:3037] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:21.468108 17612 raft_consensus.cc:2454] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5684a95375846f1bb2a9feaa078c09d in term 1.
I20260809 04:39:21.468108 17420 raft_consensus.cc:2454] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5684a95375846f1bb2a9feaa078c09d in term 1.
I20260809 04:39:21.468390 17707 leader_election.cc:302] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0833b66f20e5456ca51eaf66f705fe08, e5684a95375846f1bb2a9feaa078c09d; no voters:
I20260809 04:39:21.468559 17916 raft_consensus.cc:2781] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:21.468782 17916 raft_consensus.cc:686] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 1 LEADER]: Becoming Leader. State: Replica: e5684a95375846f1bb2a9feaa078c09d, State: Running, Role: LEADER
I20260809 04:39:21.469136 17916 consensus_queue.cc:237] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } }
I20260809 04:39:21.472355 17177 catalog_manager.cc:5495] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5684a95375846f1bb2a9feaa078c09d (127.19.92.3). New cstate: current_term: 1 leader_uuid: "e5684a95375846f1bb2a9feaa078c09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5684a95375846f1bb2a9feaa078c09d" member_type: VOTER last_known_addr { host: "127.19.92.3" port: 44835 } health_report { overall_health: HEALTHY } } }
I20260809 04:39:21.573314 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:21.569178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.19.92.3:44835)
0809 04:39:21.569465 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:21.569471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:21.569488 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:39:21.569603 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:21.569606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:21.569670 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:21.569782 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:21.569983 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.570805 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.571808 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:21.572054 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:21.572134 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:21.572429 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:21.572866 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:21.572956 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:21.573077 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 04:39:21.573133 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260809 04:39:21.579717 17420 raft_consensus.cc:1261] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08 [term 1 FOLLOWER]: Refusing update from remote peer e5684a95375846f1bb2a9feaa078c09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:21.579717 17612 raft_consensus.cc:1261] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 1 FOLLOWER]: Refusing update from remote peer e5684a95375846f1bb2a9feaa078c09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:21.580452 17916 consensus_queue.cc:1035] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0833b66f20e5456ca51eaf66f705fe08" member_type: VOTER last_known_addr { host: "127.19.92.1" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:21.580664 17920 consensus_queue.cc:1035] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6924239ada0e4feb8b38e633da5620fe" member_type: VOTER last_known_addr { host: "127.19.92.2" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:21.588456 17930 mvcc.cc:205] Tried to move back new op lower bound from 7316481481022201856 to 7316481480580259840. Current Snapshot: MvccSnapshot[applied={T|T < 7316481481022201856}]
I20260809 04:39:21.590337 17939 mvcc.cc:205] Tried to move back new op lower bound from 7316481481022201856 to 7316481480580259840. Current Snapshot: MvccSnapshot[applied={T|T < 7316481481022201856}]
I20260809 04:39:21.598269 17182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:21.598444 17182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:39:21.600811 17182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad23c69dc1a34386ba1d64d6bb768b91: Sending DeleteTablet for 3 replicas of tablet 1dd8a4c972464baaadac2efa5478b815
I20260809 04:39:21.601614 17775 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd8a4c972464baaadac2efa5478b815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:21 UTC) from {username='slave'} at 127.0.0.1:56526
I20260809 04:39:21.601703 17385 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd8a4c972464baaadac2efa5478b815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:21 UTC) from {username='slave'} at 127.0.0.1:58908
I20260809 04:39:21.601884 17582 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd8a4c972464baaadac2efa5478b815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:39:21 UTC) from {username='slave'} at 127.0.0.1:44782
I20260809 04:39:21.602380 17951 tablet_replica.cc:330] T 1dd8a4c972464baaadac2efa5478b815 P 0833b66f20e5456ca51eaf66f705fe08: stopping tablet replica
I20260809 04:39:21.602393 17952 tablet_replica.cc:330] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: stopping tablet replica
I20260809 04:39:21.602716 17952 raft_consensus.cc:2229] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:39:21.602770 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 17285
I20260809 04:39:21.620328 17950 tablet_replica.cc:330] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: stopping tablet replica
I20260809 04:39:21.620754 17950 raft_consensus.cc:2229] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:39:21.621271 17950 raft_consensus.cc:2258] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:21.627636 17950 ts_tablet_manager.cc:1905] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:21.630759 17950 ts_tablet_manager.cc:1918] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:21.630878 17950 log.cc:1196] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-2/wal/wals/1dd8a4c972464baaadac2efa5478b815
I20260809 04:39:21.631153 17950 ts_tablet_manager.cc:1939] T 1dd8a4c972464baaadac2efa5478b815 P e5684a95375846f1bb2a9feaa078c09d: Deleting consensus metadata
I20260809 04:39:21.632488 17154 catalog_manager.cc:4841] TS e5684a95375846f1bb2a9feaa078c09d (127.19.92.3:44835): tablet 1dd8a4c972464baaadac2efa5478b815 (table test-table [id=0176b7453f464e6a873b1b013b1b9ad3]) successfully deleted
I20260809 04:39:21.633958 17952 raft_consensus.cc:2258] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:39:21.635842 17952 ts_tablet_manager.cc:1905] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:39:21.638769 17952 ts_tablet_manager.cc:1918] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:39:21.638876 17952 log.cc:1196] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Deleting WAL directory at /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250097044341-19824-0/minicluster-data/ts-1/wal/wals/1dd8a4c972464baaadac2efa5478b815
I20260809 04:39:21.639215 17952 ts_tablet_manager.cc:1939] T 1dd8a4c972464baaadac2efa5478b815 P 6924239ada0e4feb8b38e633da5620fe: Deleting consensus metadata
W20260809 04:39:21.639986 17153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.92.1:43465 (error 108)
I20260809 04:39:21.640618 17153 catalog_manager.cc:4841] TS 6924239ada0e4feb8b38e633da5620fe (127.19.92.2:39675): tablet 1dd8a4c972464baaadac2efa5478b815 (table test-table [id=0176b7453f464e6a873b1b013b1b9ad3]) successfully deleted
I20260809 04:39:21.641340 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:21.640927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.1:43465 (local address 127.0.0.1:58920)
0809 04:39:21.641135 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:21.641234 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.1:43465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260809 04:39:21.641404 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 17501
W20260809 04:39:21.641595 17153 catalog_manager.cc:4584] TS 0833b66f20e5456ca51eaf66f705fe08 (127.19.92.1:43465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1dd8a4c972464baaadac2efa5478b815: Network error: Client connection negotiation failed: client connection to 127.19.92.1:43465: connect: Connection refused (error 111)
I20260809 04:39:21.658860 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 17685
I20260809 04:39:21.671224 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 17132
2026-08-09T04:39:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1584 ms)
[----------] 6 tests from EncryptionPolicyTest (12081 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:21 dist-test-slave-rmhg krb5kdc[17958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:21 dist-test-slave-rmhg krb5kdc[17958](info): set up 2 sockets
Aug 09 04:39:21 dist-test-slave-rmhg krb5kdc[17958](info): commencing operation
krb5kdc: starting...
W20260809 04:39:22.799506 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:22 dist-test-slave-rmhg krb5kdc[17958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:22.932426 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:23.045569 17973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:23.045851 17973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:23.045894 17973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:23.049927 17973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:23.049989 17973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:23.050009 17973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:23.050022 17973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:23.050036 17973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:23.055435 17973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:23.056435 17973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:23.057437 17973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:23.065686 17984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:23.065744 17982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:23.065804 17973 server_base.cc:1034] running on GCE node
W20260809 04:39:23.065693 17981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:23.066543 17973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:23.067653 17973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:23.068814 17973 hybrid_clock.cc:648] HybridClock initialized: now 1786250363068799 us; error 29 us; skew 500 ppm
Aug 09 04:39:23 dist-test-slave-rmhg krb5kdc[17958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:23.076972 17973 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:23.078004 17973 webserver.cc:460] Webserver started at http://127.0.0.1:35443/ using document root <none> and password file <none>
I20260809 04:39:23.078589 17973 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:23.078662 17973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:23.078897 17973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:23.081115 17973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "4e48c9fe368c45dbb49c47df57b91d82"
format_stamp: "Formatted at 2026-08-09 04:39:23 on dist-test-slave-rmhg"
server_key: "98d13e5cf84cca5d6d99ddd863671583"
server_key_iv: "a0ce7d8f39683f19012272fc332f4d2e"
server_key_version: "encryptionkey@0"
I20260809 04:39:23.081750 17973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "4e48c9fe368c45dbb49c47df57b91d82"
format_stamp: "Formatted at 2026-08-09 04:39:23 on dist-test-slave-rmhg"
server_key: "98d13e5cf84cca5d6d99ddd863671583"
server_key_iv: "a0ce7d8f39683f19012272fc332f4d2e"
server_key_version: "encryptionkey@0"
I20260809 04:39:23.085809 17973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:39:23.088671 17995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:23.090188 17973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:23.090337 17973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "4e48c9fe368c45dbb49c47df57b91d82"
format_stamp: "Formatted at 2026-08-09 04:39:23 on dist-test-slave-rmhg"
server_key: "98d13e5cf84cca5d6d99ddd863671583"
server_key_iv: "a0ce7d8f39683f19012272fc332f4d2e"
server_key_version: "encryptionkey@0"
I20260809 04:39:23.090456 17973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:23.104228 17973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:23.107300 17973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:23.107492 17973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:23.114427 17973 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35571
I20260809 04:39:23.114440 18078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35571 every 8 connection(s)
I20260809 04:39:23.115429 17973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:23.118366 18079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:23.120242 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 17973
I20260809 04:39:23.120371 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:23.120654 19824 external_mini_cluster.cc:1428] Setting key b2fb1476d266e07747b3f7f2494d3fa9
I20260809 04:39:23.125070 18079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82: Bootstrap starting.
I20260809 04:39:23.127822 18079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:23.128777 18079 log.cc:824] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:39:23 dist-test-slave-rmhg krb5kdc[17958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 04:39:23.131186 18079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82: No bootstrap required, opened a new log
I20260809 04:39:23.135455 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:23.122035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48623 (local address 127.0.0.1:35571)
0809 04:39:23.122453 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:23.122472 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:23.122509 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:39:23.123178 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:23.123197 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:23.123537 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:23.123817 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:23.124737 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:23.125809 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:23.127002 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:23.127512 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:23.127640 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:23.130677 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:23.130682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:23.130697 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:23.130726 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:23.132363 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:23.132939 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:23.132944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:23.132949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:23.133077 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:23.133455 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:23.133457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:23.133459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:23.133875 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:23.134048 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:23.134372 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 04:39:23.134625 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":129,"threads_started":1}
I20260809 04:39:23.135321 18079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48c9fe368c45dbb49c47df57b91d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } }
I20260809 04:39:23.135548 18079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:23.135591 18079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e48c9fe368c45dbb49c47df57b91d82, State: Initialized, Role: FOLLOWER
I20260809 04:39:23.136061 18079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48c9fe368c45dbb49c47df57b91d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } }
I20260809 04:39:23.136204 18079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:23.136271 18079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:23.136371 18079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:23.137356 18079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48c9fe368c45dbb49c47df57b91d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } }
I20260809 04:39:23.137724 18079 leader_election.cc:302] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e48c9fe368c45dbb49c47df57b91d82; no voters:
I20260809 04:39:23.138052 18079 leader_election.cc:288] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:23.138142 18084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:23.138381 18084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [term 1 LEADER]: Becoming Leader. State: Replica: 4e48c9fe368c45dbb49c47df57b91d82, State: Running, Role: LEADER
I20260809 04:39:23.138509 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:23.138796 18084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48c9fe368c45dbb49c47df57b91d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } }
I20260809 04:39:23.139330 18079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:23.140061 18084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e48c9fe368c45dbb49c47df57b91d82. Latest consensus state: current_term: 1 leader_uuid: "4e48c9fe368c45dbb49c47df57b91d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48c9fe368c45dbb49c47df57b91d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } } }
I20260809 04:39:23.140201 18084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:23.140617 18106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:23.142769 18088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e48c9fe368c45dbb49c47df57b91d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48c9fe368c45dbb49c47df57b91d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } } }
I20260809 04:39:23.142966 18088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:23.144239 18106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:23.151031 18106 catalog_manager.cc:1347] Generated new cluster ID: 3b3b8134685b4db99711c4f28f992d55
I20260809 04:39:23.151103 18106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:23.152483 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:23.142124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53705 (local address 127.0.0.1:35571)
0809 04:39:23.142235 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:23.142239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:23.142620 (+ 381us) server_negotiation.cc:407] Connection header received
0809 04:39:23.142903 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:23.142907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:23.142957 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:23.143045 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:23.144643 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:23.145423 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:23.146552 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:23.146867 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:23.146928 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:23.148405 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:23.148409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:23.148411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:23.148437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:23.150336 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:23.150885 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:23.150888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:23.150890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:23.150983 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:23.151393 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:23.151397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:23.151398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:23.151643 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:23.151764 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:23.152210 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 04:39:23.152320 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:39:23.165992 18106 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:23.167043 18106 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:23.177075 18106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e48c9fe368c45dbb49c47df57b91d82: Generated new TSK 0
I20260809 04:39:23.178102 18106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:23.190255 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 17973
W20260809 04:39:23.207656 17955 connection.cc:535] client connection to 127.0.0.1:35571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35571): Connection reset by peer (error 104)
2026-08-09T04:39:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:23 dist-test-slave-rmhg krb5kdc[18125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:23 dist-test-slave-rmhg krb5kdc[18125](info): set up 2 sockets
Aug 09 04:39:23 dist-test-slave-rmhg krb5kdc[18125](info): commencing operation
krb5kdc: starting...
W20260809 04:39:24.337981 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:24 dist-test-slave-rmhg krb5kdc[18125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:24.470571 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:24.586091 18141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:24.586395 18141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:24.586437 18141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:24.590549 18141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:24.590618 18141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:24.590637 18141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:24.590653 18141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:24.590668 18141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:24.596252 18141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:24.597545 18141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:24.598626 18141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:24.607517 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:24.607553 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:24.607568 18155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:24.607822 18141 server_base.cc:1034] running on GCE node
I20260809 04:39:24.608585 18141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:24.609889 18141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:24.611075 18141 hybrid_clock.cc:648] HybridClock initialized: now 1786250364611056 us; error 32 us; skew 500 ppm
Aug 09 04:39:24 dist-test-slave-rmhg krb5kdc[18125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:24.614212 18141 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:24.615232 18141 webserver.cc:460] Webserver started at http://127.0.0.1:45753/ using document root <none> and password file <none>
I20260809 04:39:24.615831 18141 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:24.615912 18141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:24.616147 18141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:24.618475 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "8492c5d5151b460db167170e0cbedf57"
format_stamp: "Formatted at 2026-08-09 04:39:24 on dist-test-slave-rmhg"
server_key: "e40cb94dd2b00284b0370f6870e29fd1"
server_key_iv: "867151bfcb696aa3dc4330653839c327"
server_key_version: "encryptionkey@0"
I20260809 04:39:24.619074 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "8492c5d5151b460db167170e0cbedf57"
format_stamp: "Formatted at 2026-08-09 04:39:24 on dist-test-slave-rmhg"
server_key: "e40cb94dd2b00284b0370f6870e29fd1"
server_key_iv: "867151bfcb696aa3dc4330653839c327"
server_key_version: "encryptionkey@0"
I20260809 04:39:24.623064 18141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 04:39:24.625743 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:24.627007 18141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:24.627141 18141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "8492c5d5151b460db167170e0cbedf57"
format_stamp: "Formatted at 2026-08-09 04:39:24 on dist-test-slave-rmhg"
server_key: "e40cb94dd2b00284b0370f6870e29fd1"
server_key_iv: "867151bfcb696aa3dc4330653839c327"
server_key_version: "encryptionkey@0"
I20260809 04:39:24.627250 18141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:24.652458 18141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:24.655845 18141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:24.656047 18141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:24.663412 18141 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38365
I20260809 04:39:24.663430 18235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38365 every 8 connection(s)
I20260809 04:39:24.664388 18141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:24.667202 18237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:24.668712 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 18141
I20260809 04:39:24.668855 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:24.669143 19824 external_mini_cluster.cc:1428] Setting key ce269367f89a28ae9a1d25425ac8b5fb
I20260809 04:39:24.674036 18237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57: Bootstrap starting.
I20260809 04:39:24.676712 18237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:24.677668 18237 log.cc:824] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:24.679366 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:24.670684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47643 (local address 127.0.0.1:38365)
0809 04:39:24.671146 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:24.671171 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:24.671215 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:39:24.672014 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:24.672039 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:24.672497 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:24.672901 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:24.673380 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:24.674801 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:24.675708 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:24.676388 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:24.676560 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:24.677448 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:24.677455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:24.677472 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:24.677984 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:24.678055 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:24.678091 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:24.678220 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:24.678428 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 04:39:24.678572 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":115,"threads_started":1}
I20260809 04:39:24.679886 18237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57: No bootstrap required, opened a new log
I20260809 04:39:24.682516 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:24.684566 18237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8492c5d5151b460db167170e0cbedf57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38365 } }
I20260809 04:39:24.684893 18237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:24.684998 18237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8492c5d5151b460db167170e0cbedf57, State: Initialized, Role: FOLLOWER
I20260809 04:39:24.685585 18237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8492c5d5151b460db167170e0cbedf57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38365 } }
I20260809 04:39:24.685778 18237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:24.685858 18237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:24.685966 18237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:24.687161 18237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8492c5d5151b460db167170e0cbedf57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38365 } }
I20260809 04:39:24.687557 18237 leader_election.cc:302] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8492c5d5151b460db167170e0cbedf57; no voters:
I20260809 04:39:24.687923 18237 leader_election.cc:288] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:24.688017 18251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:24.688215 18251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [term 1 LEADER]: Becoming Leader. State: Replica: 8492c5d5151b460db167170e0cbedf57, State: Running, Role: LEADER
I20260809 04:39:24.688617 18251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8492c5d5151b460db167170e0cbedf57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38365 } }
I20260809 04:39:24.689340 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:24.684886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46699 (local address 127.0.0.1:38365)
0809 04:39:24.685041 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:24.685046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:24.685288 (+ 242us) server_negotiation.cc:407] Connection header received
0809 04:39:24.685406 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:24.685409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:24.685459 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:24.685548 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:24.685793 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:24.686535 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:24.687388 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:24.687679 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:24.687778 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:24.688699 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:24.688703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:24.688706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:24.688721 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:24.688753 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:24.688766 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:24.688844 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:24.689180 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 04:39:24.689224 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:39:24.689447 18237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:24.691120 18252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8492c5d5151b460db167170e0cbedf57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8492c5d5151b460db167170e0cbedf57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38365 } } }
I20260809 04:39:24.691257 18252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:24.691623 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8492c5d5151b460db167170e0cbedf57. Latest consensus state: current_term: 1 leader_uuid: "8492c5d5151b460db167170e0cbedf57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8492c5d5151b460db167170e0cbedf57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38365 } } }
I20260809 04:39:24.692026 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57 [sys.catalog]: This master's current role is: LEADER
W20260809 04:39:24.694291 18272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:24.694363 18272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:24.694454 18267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:24.695202 18267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:24.700997 18267 catalog_manager.cc:1347] Generated new cluster ID: 287677fd7c6f4227bc18d19c322c3504
I20260809 04:39:24.701073 18267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:24.707283 18267 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:24.708165 18267 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:24.715502 18267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8492c5d5151b460db167170e0cbedf57: Generated new TSK 0
I20260809 04:39:24.716169 18267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:24.725421 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 18141
W20260809 04:39:24.735554 18123 connection.cc:535] client connection to 127.0.0.1:38365 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38365): Connection reset by peer (error 104)
2026-08-09T04:39:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1528 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:24 dist-test-slave-rmhg krb5kdc[18281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:24 dist-test-slave-rmhg krb5kdc[18281](info): set up 2 sockets
Aug 09 04:39:24 dist-test-slave-rmhg krb5kdc[18281](info): commencing operation
krb5kdc: starting...
W20260809 04:39:25.871881 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:25 dist-test-slave-rmhg krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:25.998473 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:26.112661 18298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:26.112947 18298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:26.112988 18298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:26.117036 18298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:26.117120 18298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:26.117138 18298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:26.117175 18298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:26.117195 18298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:26.123055 18298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:26.124166 18298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:26.125197 18298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:26.133805 18303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:26.133857 18306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:26.133805 18308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:26.134178 18298 server_base.cc:1034] running on GCE node
I20260809 04:39:26.134728 18298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:26.135830 18298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:26.137053 18298 hybrid_clock.cc:648] HybridClock initialized: now 1786250366137041 us; error 27 us; skew 500 ppm
Aug 09 04:39:26 dist-test-slave-rmhg krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:26.139779 18298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:26.140707 18298 webserver.cc:460] Webserver started at http://127.0.0.1:41633/ using document root <none> and password file <none>
I20260809 04:39:26.141270 18298 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:26.141337 18298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:26.141513 18298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:26.143567 18298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "734fee1bcbe54518813576e1cdea2fca"
format_stamp: "Formatted at 2026-08-09 04:39:26 on dist-test-slave-rmhg"
server_key: "9dfb19fe6d1eaae1bd392d48b88794d1"
server_key_iv: "2588720d529f18b2efaf73e78a216fe8"
server_key_version: "encryptionkey@0"
I20260809 04:39:26.144071 18298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "734fee1bcbe54518813576e1cdea2fca"
format_stamp: "Formatted at 2026-08-09 04:39:26 on dist-test-slave-rmhg"
server_key: "9dfb19fe6d1eaae1bd392d48b88794d1"
server_key_iv: "2588720d529f18b2efaf73e78a216fe8"
server_key_version: "encryptionkey@0"
I20260809 04:39:26.147818 18298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:39:26.150565 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:26.152298 18298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:39:26.152441 18298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "734fee1bcbe54518813576e1cdea2fca"
format_stamp: "Formatted at 2026-08-09 04:39:26 on dist-test-slave-rmhg"
server_key: "9dfb19fe6d1eaae1bd392d48b88794d1"
server_key_iv: "2588720d529f18b2efaf73e78a216fe8"
server_key_version: "encryptionkey@0"
I20260809 04:39:26.152575 18298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:26.157245 18298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:26.160292 18298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:26.160458 18298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:26.167217 18298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38639
I20260809 04:39:26.167229 18392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38639 every 8 connection(s)
I20260809 04:39:26.168175 18298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:26.171106 18393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:26.175246 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 18298
I20260809 04:39:26.175366 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:26.175633 19824 external_mini_cluster.cc:1428] Setting key b7d133d4473480cb9713076292adbefb
I20260809 04:39:26.177045 18393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca: Bootstrap starting.
I20260809 04:39:26.179787 18393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:26.180681 18393 log.cc:824] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:26.181209 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:26.177073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47027 (local address 127.0.0.1:38639)
0809 04:39:26.177540 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:26.177564 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:26.177607 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:39:26.178442 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:26.178467 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:26.178835 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:26.179201 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:26.179846 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:26.179851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:26.179866 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:26.179879 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:26.179943 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:26.179973 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:26.180093 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:26.180159 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 04:39:26.180302 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":116,"threads_started":1}
I20260809 04:39:26.182721 18393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca: No bootstrap required, opened a new log
I20260809 04:39:26.183895 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:26.187049 18393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fee1bcbe54518813576e1cdea2fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38639 } }
I20260809 04:39:26.187321 18393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:26.187422 18393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734fee1bcbe54518813576e1cdea2fca, State: Initialized, Role: FOLLOWER
I20260809 04:39:26.187553 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:26.186161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41303 (local address 127.0.0.1:38639)
0809 04:39:26.186270 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:26.186274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:26.186357 (+ 83us) server_negotiation.cc:407] Connection header received
0809 04:39:26.186433 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:26.186436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:26.186467 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:26.186547 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:26.187163 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:26.187166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:26.187168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:26.187180 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:26.187206 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:26.187219 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:26.187304 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:26.187392 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 04:39:26.187430 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:39:26.187958 18393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fee1bcbe54518813576e1cdea2fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38639 } }
I20260809 04:39:26.188089 18393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:26.188151 18393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:26.188246 18393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:26.189141 18393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fee1bcbe54518813576e1cdea2fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38639 } }
I20260809 04:39:26.189481 18393 leader_election.cc:302] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 734fee1bcbe54518813576e1cdea2fca; no voters:
I20260809 04:39:26.189774 18393 leader_election.cc:288] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:26.189852 18410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:26.190076 18410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [term 1 LEADER]: Becoming Leader. State: Replica: 734fee1bcbe54518813576e1cdea2fca, State: Running, Role: LEADER
I20260809 04:39:26.190395 18410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fee1bcbe54518813576e1cdea2fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38639 } }
I20260809 04:39:26.190971 18393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:26.192196 18411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "734fee1bcbe54518813576e1cdea2fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fee1bcbe54518813576e1cdea2fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38639 } } }
I20260809 04:39:26.192319 18411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:26.192747 18431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:26.192723 18414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 734fee1bcbe54518813576e1cdea2fca. Latest consensus state: current_term: 1 leader_uuid: "734fee1bcbe54518813576e1cdea2fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fee1bcbe54518813576e1cdea2fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38639 } } }
I20260809 04:39:26.192868 18414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:26.195626 18431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:26.201366 18431 catalog_manager.cc:1347] Generated new cluster ID: e62b45bafe064c2da5990f07121ab639
I20260809 04:39:26.201426 18431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:26.209776 18431 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:26.210109 18431 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:26.220108 18431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 734fee1bcbe54518813576e1cdea2fca: Generated new TSK 0
I20260809 04:39:26.220871 18431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:26.229754 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 18298
2026-08-09T04:39:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1510 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:26 dist-test-slave-rmhg krb5kdc[18446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:26 dist-test-slave-rmhg krb5kdc[18446](info): set up 2 sockets
Aug 09 04:39:26 dist-test-slave-rmhg krb5kdc[18446](info): commencing operation
krb5kdc: starting...
W20260809 04:39:27.369061 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:27 dist-test-slave-rmhg krb5kdc[18446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:27.495606 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:27.616391 18468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:27.616704 18468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:27.616770 18468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:27.620910 18468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:27.620993 18468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:27.621021 18468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:27.621043 18468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:27.621073 18468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:27.626685 18468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:27.627808 18468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:27.628854 18468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:27.637292 18483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:27.637300 18477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:27.637292 18476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:27.637620 18468 server_base.cc:1034] running on GCE node
I20260809 04:39:27.638177 18468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:27.639289 18468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:27.640448 18468 hybrid_clock.cc:648] HybridClock initialized: now 1786250367640428 us; error 33 us; skew 500 ppm
Aug 09 04:39:27 dist-test-slave-rmhg krb5kdc[18446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:27.643091 18468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:27.644006 18468 webserver.cc:460] Webserver started at http://127.0.0.1:35241/ using document root <none> and password file <none>
I20260809 04:39:27.644572 18468 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:27.644641 18468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:27.644857 18468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:27.646993 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "aac77ebac51a41a8ad17780424cc52ac"
format_stamp: "Formatted at 2026-08-09 04:39:27 on dist-test-slave-rmhg"
server_key: "e03d21624abf5e1a14e59a7e4dcd7e5a"
server_key_iv: "ab3dc6feab99f06c5a7c3c0f0fa95939"
server_key_version: "encryptionkey@0"
I20260809 04:39:27.647588 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "aac77ebac51a41a8ad17780424cc52ac"
format_stamp: "Formatted at 2026-08-09 04:39:27 on dist-test-slave-rmhg"
server_key: "e03d21624abf5e1a14e59a7e4dcd7e5a"
server_key_iv: "ab3dc6feab99f06c5a7c3c0f0fa95939"
server_key_version: "encryptionkey@0"
I20260809 04:39:27.651604 18468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:39:27.654142 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:27.655395 18468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:27.655539 18468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "aac77ebac51a41a8ad17780424cc52ac"
format_stamp: "Formatted at 2026-08-09 04:39:27 on dist-test-slave-rmhg"
server_key: "e03d21624abf5e1a14e59a7e4dcd7e5a"
server_key_iv: "ab3dc6feab99f06c5a7c3c0f0fa95939"
server_key_version: "encryptionkey@0"
I20260809 04:39:27.655642 18468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:27.670161 18468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:27.673022 18468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:27.673229 18468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:27.679888 18468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40825
I20260809 04:39:27.679932 18575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40825 every 8 connection(s)
I20260809 04:39:27.680824 18468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:27.682961 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 18468
I20260809 04:39:27.683075 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:27.683341 19824 external_mini_cluster.cc:1428] Setting key ca170b48609574303ecfb05467e75470
I20260809 04:39:27.683727 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:27.691052 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac: Bootstrap starting.
Aug 09 04:39:27 dist-test-slave-rmhg krb5kdc[18446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 04:39:27.693511 18577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:27.694216 18577 log.cc:824] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:27.696410 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac: No bootstrap required, opened a new log
I20260809 04:39:27.697525 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:27.684709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41247 (local address 127.0.0.1:40825)
0809 04:39:27.685129 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:27.685168 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:27.685213 (+ 45us) server_negotiation.cc:407] Connection header received
0809 04:39:27.685974 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:27.685998 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:27.686444 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:27.686830 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:27.687484 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:27.688997 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:27.689912 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:27.690626 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:27.690785 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:27.693124 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:27.693131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:27.693167 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:27.693200 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:27.694825 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:27.695240 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:27.695245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:27.695249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:27.695353 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:27.695624 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:27.695627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:27.695629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:27.696019 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:27.696191 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:27.696482 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 04:39:27.696767 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260809 04:39:27.700187 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:27.700124 18577 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac77ebac51a41a8ad17780424cc52ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } }
I20260809 04:39:27.700390 18577 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:27.700451 18577 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac77ebac51a41a8ad17780424cc52ac, State: Initialized, Role: FOLLOWER
I20260809 04:39:27.700898 18577 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac77ebac51a41a8ad17780424cc52ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } }
I20260809 04:39:27.701038 18577 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:27.701107 18577 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:27.701228 18577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:27.702260 18577 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac77ebac51a41a8ad17780424cc52ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } }
I20260809 04:39:27.702713 18577 leader_election.cc:302] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac77ebac51a41a8ad17780424cc52ac; no voters:
I20260809 04:39:27.703058 18577 leader_election.cc:288] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:27.703145 18591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:27.703322 18591 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [term 1 LEADER]: Becoming Leader. State: Replica: aac77ebac51a41a8ad17780424cc52ac, State: Running, Role: LEADER
I20260809 04:39:27.703686 18591 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac77ebac51a41a8ad17780424cc52ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } }
I20260809 04:39:27.704432 18577 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:27.705374 18594 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac77ebac51a41a8ad17780424cc52ac. Latest consensus state: current_term: 1 leader_uuid: "aac77ebac51a41a8ad17780424cc52ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac77ebac51a41a8ad17780424cc52ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } } }
I20260809 04:39:27.705511 18594 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:27.705902 18595 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac77ebac51a41a8ad17780424cc52ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac77ebac51a41a8ad17780424cc52ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } } }
I20260809 04:39:27.706077 18595 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:27.707060 18606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:27.710234 18606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:27.710944 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:27.702305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53551 (local address 127.0.0.1:40825)
0809 04:39:27.702411 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:27.702415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:27.702644 (+ 229us) server_negotiation.cc:407] Connection header received
0809 04:39:27.702779 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:27.702781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:27.702819 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:27.702896 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:27.703746 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:27.704380 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:27.705441 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:27.705782 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:27.705857 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:27.707609 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:27.707612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:27.707614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:27.707637 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:27.709096 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:27.709603 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:27.709606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:27.709608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:27.709697 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:27.709977 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:27.709981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:27.709983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:27.710227 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:27.710355 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:27.710672 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 04:39:27.710796 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 04:39:27.716179 18606 catalog_manager.cc:1347] Generated new cluster ID: 4e978e2da6884c43961bc95910a1236f
I20260809 04:39:27.716245 18606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:27.734697 18606 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:27.735653 18606 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:27.741707 18606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac77ebac51a41a8ad17780424cc52ac: Generated new TSK 0
I20260809 04:39:27.742411 18606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:27.750751 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 18468
2026-08-09T04:39:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:27 dist-test-slave-rmhg krb5kdc[18626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:27 dist-test-slave-rmhg krb5kdc[18626](info): set up 2 sockets
Aug 09 04:39:27 dist-test-slave-rmhg krb5kdc[18626](info): commencing operation
krb5kdc: starting...
W20260809 04:39:28.890969 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:28 dist-test-slave-rmhg krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:29.014510 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:29.127439 18648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:29.127868 18648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:29.127974 18648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:29.132141 18648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:29.132205 18648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:29.132227 18648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:29.132258 18648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:29.132285 18648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:29.137651 18648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:29.138705 18648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:29.139731 18648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:29.148420 18654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:29.148416 18659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:29.148451 18653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:29.148645 18648 server_base.cc:1034] running on GCE node
I20260809 04:39:29.149303 18648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:29.150373 18648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:29.151536 18648 hybrid_clock.cc:648] HybridClock initialized: now 1786250369151517 us; error 32 us; skew 500 ppm
Aug 09 04:39:29 dist-test-slave-rmhg krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:29.157768 18648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:29.158800 18648 webserver.cc:460] Webserver started at http://127.0.0.1:43627/ using document root <none> and password file <none>
I20260809 04:39:29.159444 18648 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:29.159544 18648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:29.159898 18648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:29.162555 18648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "cd006f908db040d6b65e7d3c82dae170"
format_stamp: "Formatted at 2026-08-09 04:39:29 on dist-test-slave-rmhg"
server_key: "46402956661517cfa0c1022adba5926e"
server_key_iv: "c4a0557a701b0b864ce31ba529191406"
server_key_version: "encryptionkey@0"
I20260809 04:39:29.163127 18648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "cd006f908db040d6b65e7d3c82dae170"
format_stamp: "Formatted at 2026-08-09 04:39:29 on dist-test-slave-rmhg"
server_key: "46402956661517cfa0c1022adba5926e"
server_key_iv: "c4a0557a701b0b864ce31ba529191406"
server_key_version: "encryptionkey@0"
I20260809 04:39:29.167073 18648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:39:29.169661 18668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:29.170940 18648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:29.171074 18648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "cd006f908db040d6b65e7d3c82dae170"
format_stamp: "Formatted at 2026-08-09 04:39:29 on dist-test-slave-rmhg"
server_key: "46402956661517cfa0c1022adba5926e"
server_key_iv: "c4a0557a701b0b864ce31ba529191406"
server_key_version: "encryptionkey@0"
I20260809 04:39:29.171182 18648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:29.194494 18648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:29.197606 18648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:29.197786 18648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:29.204875 18648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43077
I20260809 04:39:29.204890 18743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43077 every 8 connection(s)
I20260809 04:39:29.205885 18648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:29.208897 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:29.212188 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 18648
I20260809 04:39:29.212339 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:29.212663 19824 external_mini_cluster.cc:1428] Setting key 6c6a037c4c3f3de58aeb2800f18fb844
I20260809 04:39:29.214794 18744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170: Bootstrap starting.
I20260809 04:39:29.217033 18744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:29.217763 18744 log.cc:824] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:29.219658 18744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170: No bootstrap required, opened a new log
I20260809 04:39:29.221282 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:29.214064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37859 (local address 127.0.0.1:43077)
0809 04:39:29.214479 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:29.214495 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:29.214530 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:39:29.215169 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:29.215188 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:29.215541 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:29.215811 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:29.216136 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:29.217118 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:29.218076 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:29.218565 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:29.218737 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:29.219512 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:29.219517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:29.219532 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:29.219949 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:29.220009 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:29.220033 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:29.220118 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:29.220261 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 04:39:29.220480 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260809 04:39:29.223726 18744 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006f908db040d6b65e7d3c82dae170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43077 } }
I20260809 04:39:29.223948 18744 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:29.224025 18744 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd006f908db040d6b65e7d3c82dae170, State: Initialized, Role: FOLLOWER
I20260809 04:39:29.224215 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:29.224483 18744 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006f908db040d6b65e7d3c82dae170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43077 } }
I20260809 04:39:29.224617 18744 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:29.224679 18744 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:29.224776 18744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:29.225793 18744 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006f908db040d6b65e7d3c82dae170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43077 } }
I20260809 04:39:29.226151 18744 leader_election.cc:302] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd006f908db040d6b65e7d3c82dae170; no voters:
I20260809 04:39:29.226491 18744 leader_election.cc:288] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:29.226575 18760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:29.226779 18760 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [term 1 LEADER]: Becoming Leader. State: Replica: cd006f908db040d6b65e7d3c82dae170, State: Running, Role: LEADER
I20260809 04:39:29.227098 18760 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006f908db040d6b65e7d3c82dae170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43077 } }
I20260809 04:39:29.227794 18744 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:29.228765 18763 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd006f908db040d6b65e7d3c82dae170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006f908db040d6b65e7d3c82dae170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43077 } } }
I20260809 04:39:29.228894 18763 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:29.229418 18762 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd006f908db040d6b65e7d3c82dae170. Latest consensus state: current_term: 1 leader_uuid: "cd006f908db040d6b65e7d3c82dae170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006f908db040d6b65e7d3c82dae170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43077 } } }
I20260809 04:39:29.229591 18762 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:29.231069 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:29.226325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43455 (local address 127.0.0.1:43077)
0809 04:39:29.226435 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:29.226439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:29.226719 (+ 280us) server_negotiation.cc:407] Connection header received
0809 04:39:29.226874 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:29.226877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:29.226923 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:29.226984 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:29.227326 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:29.228086 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:29.229190 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:29.229498 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:29.229998 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:29.230427 (+ 429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:29.230431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:29.230434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:29.230448 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:29.230478 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:29.230493 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:29.230851 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:29.230882 (+ 31us) server_negotiation.cc:299] Negotiation successful
0809 04:39:29.230943 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260809 04:39:29.233173 18787 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:29.233250 18787 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:29.233444 18777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:29.234311 18777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:29.240171 18777 catalog_manager.cc:1347] Generated new cluster ID: d151a94ef5e9482fbb4022991a91278e
I20260809 04:39:29.240242 18777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:29.260592 18777 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:29.262010 18777 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:29.275481 18777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd006f908db040d6b65e7d3c82dae170: Generated new TSK 0
I20260809 04:39:29.276371 18777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:29.283056 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 18648
2026-08-09T04:39:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1530 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:29 dist-test-slave-rmhg krb5kdc[18799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:29 dist-test-slave-rmhg krb5kdc[18799](info): set up 2 sockets
Aug 09 04:39:29 dist-test-slave-rmhg krb5kdc[18799](info): commencing operation
krb5kdc: starting...
W20260809 04:39:30.420459 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:30 dist-test-slave-rmhg krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:30.550071 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:30.663161 18817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:30.663457 18817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:30.663520 18817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:30.667518 18817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:30.667587 18817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:30.667618 18817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:30.667639 18817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:30.667667 18817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:30.672992 18817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:30.674063 18817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:30.675084 18817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:30.683686 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:30.683694 18826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:30.683704 18829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:30.683941 18817 server_base.cc:1034] running on GCE node
I20260809 04:39:30.684569 18817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:30.685704 18817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:30.686880 18817 hybrid_clock.cc:648] HybridClock initialized: now 1786250370686859 us; error 33 us; skew 500 ppm
Aug 09 04:39:30 dist-test-slave-rmhg krb5kdc[18799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:30.689594 18817 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:30.690464 18817 webserver.cc:460] Webserver started at http://127.0.0.1:36449/ using document root <none> and password file <none>
I20260809 04:39:30.690991 18817 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:30.691049 18817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:30.691215 18817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:30.693266 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "8933ae93aa8440ffbcaa2114abe2a0f3"
format_stamp: "Formatted at 2026-08-09 04:39:30 on dist-test-slave-rmhg"
server_key: "5b1b0c0dbf9ece45d5c6460ae383afa3"
server_key_iv: "43eb2a81dea041e53a77856aa2d5149e"
server_key_version: "encryptionkey@0"
I20260809 04:39:30.693773 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "8933ae93aa8440ffbcaa2114abe2a0f3"
format_stamp: "Formatted at 2026-08-09 04:39:30 on dist-test-slave-rmhg"
server_key: "5b1b0c0dbf9ece45d5c6460ae383afa3"
server_key_iv: "43eb2a81dea041e53a77856aa2d5149e"
server_key_version: "encryptionkey@0"
I20260809 04:39:30.697399 18817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:30.699769 18837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:30.701025 18817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:30.701143 18817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "8933ae93aa8440ffbcaa2114abe2a0f3"
format_stamp: "Formatted at 2026-08-09 04:39:30 on dist-test-slave-rmhg"
server_key: "5b1b0c0dbf9ece45d5c6460ae383afa3"
server_key_iv: "43eb2a81dea041e53a77856aa2d5149e"
server_key_version: "encryptionkey@0"
I20260809 04:39:30.701274 18817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:30.705767 18817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:30.708472 18817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:30.708642 18817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:30.715204 18817 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34863
I20260809 04:39:30.715248 18908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34863 every 8 connection(s)
I20260809 04:39:30.716096 18817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:30.717346 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 18817
I20260809 04:39:30.717499 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:30.717826 19824 external_mini_cluster.cc:1428] Setting key 7131262795b4e46fffec6c20c9a98589
I20260809 04:39:30.719475 18909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:30.722960 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:30.719245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50741 (local address 127.0.0.1:34863)
0809 04:39:30.719709 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:30.719726 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:30.719761 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:39:30.720420 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:30.720438 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:30.720723 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:30.721039 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:30.721825 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:30.721830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:30.721841 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:30.721851 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:30.721905 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:30.721925 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:30.722062 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:30.722131 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 04:39:30.722283 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260809 04:39:30.725524 18909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3: Bootstrap starting.
I20260809 04:39:30.725658 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:30.727905 18909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:30.728699 18909 log.cc:824] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:30.729103 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:30.727525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42537 (local address 127.0.0.1:34863)
0809 04:39:30.727617 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:30.727621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:30.727837 (+ 216us) server_negotiation.cc:407] Connection header received
0809 04:39:30.727941 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:30.727944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:30.727978 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:30.728047 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:30.728735 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:30.728739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:30.728742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:30.728753 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:30.728779 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:30.728790 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:30.728863 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:30.728962 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 04:39:30.728998 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 04:39:30.730675 18909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3: No bootstrap required, opened a new log
I20260809 04:39:30.734200 18909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8933ae93aa8440ffbcaa2114abe2a0f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } }
I20260809 04:39:30.734411 18909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:30.734486 18909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8933ae93aa8440ffbcaa2114abe2a0f3, State: Initialized, Role: FOLLOWER
I20260809 04:39:30.734905 18909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8933ae93aa8440ffbcaa2114abe2a0f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } }
I20260809 04:39:30.735059 18909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:30.735134 18909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:30.735244 18909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:30.736138 18909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8933ae93aa8440ffbcaa2114abe2a0f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } }
I20260809 04:39:30.736461 18909 leader_election.cc:302] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8933ae93aa8440ffbcaa2114abe2a0f3; no voters:
I20260809 04:39:30.736742 18909 leader_election.cc:288] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:30.736855 18927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:30.737058 18927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [term 1 LEADER]: Becoming Leader. State: Replica: 8933ae93aa8440ffbcaa2114abe2a0f3, State: Running, Role: LEADER
I20260809 04:39:30.737387 18927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8933ae93aa8440ffbcaa2114abe2a0f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } }
I20260809 04:39:30.737887 18909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:30.738897 18929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8933ae93aa8440ffbcaa2114abe2a0f3. Latest consensus state: current_term: 1 leader_uuid: "8933ae93aa8440ffbcaa2114abe2a0f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8933ae93aa8440ffbcaa2114abe2a0f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } } }
I20260809 04:39:30.739063 18929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:30.738890 18928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8933ae93aa8440ffbcaa2114abe2a0f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8933ae93aa8440ffbcaa2114abe2a0f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } } }
I20260809 04:39:30.739182 18928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:30.739601 18942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:30.742683 18942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:30.748099 18942 catalog_manager.cc:1347] Generated new cluster ID: cfdb1a36769641c08e5c53760202c1ff
I20260809 04:39:30.748157 18942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:30.768385 18942 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:30.768738 18942 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:30.775135 18942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8933ae93aa8440ffbcaa2114abe2a0f3: Generated new TSK 0
I20260809 04:39:30.775847 18942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:30.783017 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 18817
2026-08-09T04:39:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:30 dist-test-slave-rmhg krb5kdc[18960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:30 dist-test-slave-rmhg krb5kdc[18960](info): set up 2 sockets
Aug 09 04:39:30 dist-test-slave-rmhg krb5kdc[18960](info): commencing operation
krb5kdc: starting...
W20260809 04:39:31.910480 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:31 dist-test-slave-rmhg krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:32.031885 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:32.142666 18979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:32.142967 18979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:32.143029 18979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:32.147064 18979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:32.147135 18979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:32.147162 18979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:32.147183 18979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:32.147204 18979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:32.152616 18979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:32.153796 18979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:32.154862 18979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:39:32.163733 18979 server_base.cc:1034] running on GCE node
W20260809 04:39:32.163861 18991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:32.163767 18985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:32.163661 18987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:32.164597 18979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:32.165863 18979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:32.167047 18979 hybrid_clock.cc:648] HybridClock initialized: now 1786250372167027 us; error 30 us; skew 500 ppm
Aug 09 04:39:32 dist-test-slave-rmhg krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:32.170488 18979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:32.171715 18979 webserver.cc:460] Webserver started at http://127.0.0.1:34603/ using document root <none> and password file <none>
I20260809 04:39:32.172380 18979 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:32.172461 18979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:32.172722 18979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:32.174896 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4"
format_stamp: "Formatted at 2026-08-09 04:39:32 on dist-test-slave-rmhg"
server_key: "3019b0aaa8f6b803f99818587fce1102"
server_key_iv: "8f7e3349de2b6cdb54e108849ce19f25"
server_key_version: "encryptionkey@0"
I20260809 04:39:32.175457 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4"
format_stamp: "Formatted at 2026-08-09 04:39:32 on dist-test-slave-rmhg"
server_key: "3019b0aaa8f6b803f99818587fce1102"
server_key_iv: "8f7e3349de2b6cdb54e108849ce19f25"
server_key_version: "encryptionkey@0"
I20260809 04:39:32.179275 18979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 04:39:32.181800 18998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:32.183017 18979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:32.183128 18979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4"
format_stamp: "Formatted at 2026-08-09 04:39:32 on dist-test-slave-rmhg"
server_key: "3019b0aaa8f6b803f99818587fce1102"
server_key_iv: "8f7e3349de2b6cdb54e108849ce19f25"
server_key_version: "encryptionkey@0"
I20260809 04:39:32.183228 18979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:32.210120 18979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:32.213326 18979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:32.213510 18979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:32.220306 18979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45479
I20260809 04:39:32.220335 19077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45479 every 8 connection(s)
I20260809 04:39:32.221436 18979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:32.224213 19078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:32.228973 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 18979
I20260809 04:39:32.229101 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:32.229380 19824 external_mini_cluster.cc:1428] Setting key 1a339a8082dc9229d3b2327255e43b28
I20260809 04:39:32.230052 19078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4: Bootstrap starting.
I20260809 04:39:32.232776 19078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:32.233716 19078 log.cc:824] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:32.236027 19078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4: No bootstrap required, opened a new log
Aug 09 04:39:32 dist-test-slave-rmhg krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 04:39:32.240350 19078 raft_consensus.cc:348] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } }
I20260809 04:39:32.240653 19078 raft_consensus.cc:374] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:32.240731 19078 raft_consensus.cc:729] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54ba7163c9d4f3b8cdccd154bfa28d4, State: Initialized, Role: FOLLOWER
I20260809 04:39:32.241321 19078 consensus_queue.cc:260] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } }
I20260809 04:39:32.241500 19078 raft_consensus.cc:388] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:32.241570 19078 raft_consensus.cc:482] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:32.241678 19078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:32.242820 19078 raft_consensus.cc:504] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } }
I20260809 04:39:32.243223 19078 leader_election.cc:302] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f54ba7163c9d4f3b8cdccd154bfa28d4; no voters:
I20260809 04:39:32.243623 19078 leader_election.cc:288] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:32.243722 19085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:32.243954 19085 raft_consensus.cc:686] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [term 1 LEADER]: Becoming Leader. State: Replica: f54ba7163c9d4f3b8cdccd154bfa28d4, State: Running, Role: LEADER
I20260809 04:39:32.244374 19085 consensus_queue.cc:237] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } }
I20260809 04:39:32.245054 19078 sys_catalog.cc:564] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:32.246256 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:32.230934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:46097 (local address 127.0.0.1:45479)
0809 04:39:32.231385 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:32.231405 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:32.231459 (+ 54us) server_negotiation.cc:407] Connection header received
0809 04:39:32.232329 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:32.232353 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:32.232816 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:32.233226 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:32.234052 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:32.235413 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:32.236445 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:32.237145 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:32.237353 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:32.240177 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:32.240183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:32.240201 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:32.240237 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:32.242257 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:32.242809 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:32.242815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:32.242822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:32.242981 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:32.243352 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:32.243356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:32.243358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:32.243930 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:32.244227 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:32.244938 (+ 711us) server_negotiation.cc:299] Negotiation successful
0809 04:39:32.245215 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":127,"threads_started":1}
I20260809 04:39:32.248133 19087 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f54ba7163c9d4f3b8cdccd154bfa28d4. Latest consensus state: current_term: 1 leader_uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } } }
I20260809 04:39:32.248188 19086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54ba7163c9d4f3b8cdccd154bfa28d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } } }
I20260809 04:39:32.248266 19087 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:32.248283 19086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:32.249801 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260809 04:39:32.252398 19106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:32.252482 19106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:32.252584 19108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:32.253396 19108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:32.261667 19108 catalog_manager.cc:1347] Generated new cluster ID: ff9d08b57fef4eb8b845ecbe4a471700
I20260809 04:39:32.261740 19108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:32.263080 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:32.252111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:51675 (local address 127.0.0.1:45479)
0809 04:39:32.252224 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:32.252229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:32.255245 (+ 3016us) server_negotiation.cc:407] Connection header received
0809 04:39:32.255365 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:32.255368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:32.255424 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:32.255490 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:32.256189 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:32.256744 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:32.257727 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:32.257986 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:32.258072 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:32.259483 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:32.259486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:32.259489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:32.259512 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:32.260879 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:32.261484 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:32.261488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:32.261490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:32.261577 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:32.262094 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:32.262098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:32.262100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:32.262337 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:32.262496 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:32.262798 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 04:39:32.262910 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:39:32.270304 19108 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:32.272032 19108 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:32.297092 19108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f54ba7163c9d4f3b8cdccd154bfa28d4: Generated new TSK 0
I20260809 04:39:32.298036 19108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:32.309060 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 18979
W20260809 04:39:32.320479 18957 connection.cc:535] client connection to 127.0.0.1:45479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45479): Connection reset by peer (error 104)
2026-08-09T04:39:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1521 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:32 dist-test-slave-rmhg krb5kdc[19121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:32 dist-test-slave-rmhg krb5kdc[19121](info): set up 2 sockets
Aug 09 04:39:32 dist-test-slave-rmhg krb5kdc[19121](info): commencing operation
krb5kdc: starting...
W20260809 04:39:33.430465 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:33 dist-test-slave-rmhg krb5kdc[19121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:33.559533 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:33.674314 19143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:33.674579 19143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:33.674635 19143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:33.678602 19143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:33.678665 19143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:33.678691 19143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:33.678714 19143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:33.678732 19143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:33.684049 19143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:33.685194 19143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:33.686231 19143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:33.694530 19154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:33.694590 19143 server_base.cc:1034] running on GCE node
W20260809 04:39:33.694530 19150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:33.694562 19151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:33.695251 19143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:33.696365 19143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:33.697556 19143 hybrid_clock.cc:648] HybridClock initialized: now 1786250373697540 us; error 27 us; skew 500 ppm
Aug 09 04:39:33 dist-test-slave-rmhg krb5kdc[19121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:33.700242 19143 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:33.701195 19143 webserver.cc:460] Webserver started at http://127.0.0.1:35283/ using document root <none> and password file <none>
I20260809 04:39:33.701761 19143 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:33.701829 19143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:33.702021 19143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:33.704056 19143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "d7c96cb4252f408299162d004326aa25"
format_stamp: "Formatted at 2026-08-09 04:39:33 on dist-test-slave-rmhg"
server_key: "5ba3b3ea2b12bfc8d5b49589f771f4e2"
server_key_iv: "4c31d2264333734c956c17d6cca3b206"
server_key_version: "encryptionkey@0"
I20260809 04:39:33.704619 19143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "d7c96cb4252f408299162d004326aa25"
format_stamp: "Formatted at 2026-08-09 04:39:33 on dist-test-slave-rmhg"
server_key: "5ba3b3ea2b12bfc8d5b49589f771f4e2"
server_key_iv: "4c31d2264333734c956c17d6cca3b206"
server_key_version: "encryptionkey@0"
I20260809 04:39:33.708303 19143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:39:33.710754 19170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:33.711995 19143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:39:33.712119 19143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "d7c96cb4252f408299162d004326aa25"
format_stamp: "Formatted at 2026-08-09 04:39:33 on dist-test-slave-rmhg"
server_key: "5ba3b3ea2b12bfc8d5b49589f771f4e2"
server_key_iv: "4c31d2264333734c956c17d6cca3b206"
server_key_version: "encryptionkey@0"
I20260809 04:39:33.712224 19143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:33.727370 19143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:33.730404 19143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:33.730597 19143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:33.737578 19143 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44339
I20260809 04:39:33.737598 19234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44339 every 8 connection(s)
I20260809 04:39:33.738543 19143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:33.741286 19235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:33.746958 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 19143
I20260809 04:39:33.746978 19235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25: Bootstrap starting.
I20260809 04:39:33.747123 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:33.747432 19824 external_mini_cluster.cc:1428] Setting key 718999c0013895e2ff9ebfa3dd5bdec8
I20260809 04:39:33.749611 19235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:33.750312 19235 log.cc:824] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:33.752326 19235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25: No bootstrap required, opened a new log
I20260809 04:39:33.756357 19235 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c96cb4252f408299162d004326aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } }
I20260809 04:39:33.756592 19235 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:33.756650 19235 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c96cb4252f408299162d004326aa25, State: Initialized, Role: FOLLOWER
I20260809 04:39:33.756667 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:33.748760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:60887 (local address 127.0.0.1:44339)
0809 04:39:33.749210 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:33.749227 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:33.749263 (+ 36us) server_negotiation.cc:407] Connection header received
0809 04:39:33.749981 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:33.750001 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:33.750359 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:33.750687 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:33.751138 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:33.752134 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:33.753379 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:33.753892 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:33.754024 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:33.754902 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:33.754908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:33.754924 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:33.755297 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:33.755353 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:33.755375 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:33.755495 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:33.755700 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 04:39:33.755848 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":140,"threads_started":1}
I20260809 04:39:33.757126 19235 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c96cb4252f408299162d004326aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } }
I20260809 04:39:33.757308 19235 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:33.757372 19235 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:33.757462 19235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:33.758373 19235 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c96cb4252f408299162d004326aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } }
I20260809 04:39:33.758739 19235 leader_election.cc:302] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7c96cb4252f408299162d004326aa25; no voters:
I20260809 04:39:33.759085 19235 leader_election.cc:288] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:33.759171 19240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:33.759380 19240 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [term 1 LEADER]: Becoming Leader. State: Replica: d7c96cb4252f408299162d004326aa25, State: Running, Role: LEADER
I20260809 04:39:33.759756 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:33.759783 19240 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c96cb4252f408299162d004326aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } }
I20260809 04:39:33.760306 19235 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:33.761096 19240 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c96cb4252f408299162d004326aa25. Latest consensus state: current_term: 1 leader_uuid: "d7c96cb4252f408299162d004326aa25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c96cb4252f408299162d004326aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } } }
I20260809 04:39:33.761269 19240 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:33.761541 19242 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7c96cb4252f408299162d004326aa25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c96cb4252f408299162d004326aa25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44339 } } }
I20260809 04:39:33.761698 19242 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:33.761718 19259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:33.765383 19259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:33.767930 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:33.762326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:37163 (local address 127.0.0.1:44339)
0809 04:39:33.762443 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:33.762447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:33.762973 (+ 526us) server_negotiation.cc:407] Connection header received
0809 04:39:33.763114 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:33.763117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:33.763183 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:33.763267 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:33.763999 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:33.764767 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:33.766024 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:33.766365 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:33.766483 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:33.767422 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:33.767427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:33.767430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:33.767444 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:33.767474 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:33.767488 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:33.767579 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:33.767756 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 04:39:33.767794 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 04:39:33.771086 19259 catalog_manager.cc:1347] Generated new cluster ID: c80221a1b4f44244899ecbfd4619961a
I20260809 04:39:33.771159 19259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:33.788753 19259 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:33.789816 19259 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:33.801301 19259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7c96cb4252f408299162d004326aa25: Generated new TSK 0
I20260809 04:39:33.802034 19259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:33.805580 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 19143
W20260809 04:39:33.820466 19119 connection.cc:535] client connection to 127.0.0.1:44339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44339): Connection reset by peer (error 104)
2026-08-09T04:39:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:33 dist-test-slave-rmhg krb5kdc[19278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:33 dist-test-slave-rmhg krb5kdc[19278](info): set up 2 sockets
Aug 09 04:39:33 dist-test-slave-rmhg krb5kdc[19278](info): commencing operation
krb5kdc: starting...
W20260809 04:39:34.939885 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:35 dist-test-slave-rmhg krb5kdc[19278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:35.066617 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:35.179035 19294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:35.179343 19294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:35.179406 19294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:35.184297 19294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:35.184378 19294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:35.184424 19294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:35.184449 19294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:35.184479 19294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:35.190109 19294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:35.191241 19294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:35.192284 19294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:35.200872 19307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:35.200883 19303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:35.200891 19304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:35.200888 19294 server_base.cc:1034] running on GCE node
I20260809 04:39:35.201648 19294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:35.202674 19294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:35.203819 19294 hybrid_clock.cc:648] HybridClock initialized: now 1786250375203803 us; error 28 us; skew 500 ppm
Aug 09 04:39:35 dist-test-slave-rmhg krb5kdc[19278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:35.209985 19294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:35.210945 19294 webserver.cc:460] Webserver started at http://127.0.0.1:45291/ using document root <none> and password file <none>
I20260809 04:39:35.211508 19294 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:35.211575 19294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:35.211781 19294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:35.213938 19294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "a78e19b102fe46c0a55235b6c82ebf56"
format_stamp: "Formatted at 2026-08-09 04:39:35 on dist-test-slave-rmhg"
server_key: "9edffc52ce0ce67bef631f1dde31fcde"
server_key_iv: "eb58da647b890bd347555326707a9a6e"
server_key_version: "encryptionkey@0"
I20260809 04:39:35.214457 19294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "a78e19b102fe46c0a55235b6c82ebf56"
format_stamp: "Formatted at 2026-08-09 04:39:35 on dist-test-slave-rmhg"
server_key: "9edffc52ce0ce67bef631f1dde31fcde"
server_key_iv: "eb58da647b890bd347555326707a9a6e"
server_key_version: "encryptionkey@0"
I20260809 04:39:35.218094 19294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:39:35.220588 19317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:35.222054 19294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:39:35.222185 19294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "a78e19b102fe46c0a55235b6c82ebf56"
format_stamp: "Formatted at 2026-08-09 04:39:35 on dist-test-slave-rmhg"
server_key: "9edffc52ce0ce67bef631f1dde31fcde"
server_key_iv: "eb58da647b890bd347555326707a9a6e"
server_key_version: "encryptionkey@0"
I20260809 04:39:35.222290 19294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:35.226886 19294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:35.229620 19294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:35.229782 19294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:35.236709 19294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42275
I20260809 04:39:35.236738 19395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42275 every 8 connection(s)
I20260809 04:39:35.237684 19294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:35.240571 19396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:35.243772 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 19294
I20260809 04:39:35.243929 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:35.244238 19824 external_mini_cluster.cc:1428] Setting key b4f5d678e426cc51c5493537f41bd6f4
I20260809 04:39:35.246910 19396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56: Bootstrap starting.
I20260809 04:39:35.249552 19396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:35.249703 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:35.245712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:50909 (local address 127.0.0.1:42275)
0809 04:39:35.246136 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:35.246153 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:35.246189 (+ 36us) server_negotiation.cc:407] Connection header received
0809 04:39:35.246834 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:35.246852 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:35.247178 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:35.247464 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:35.248340 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:35.248346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:35.248358 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:35.248370 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:35.248432 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:35.248452 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:35.248581 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:35.248681 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 04:39:35.248845 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260809 04:39:35.250330 19396 log.cc:824] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:35.252373 19396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56: No bootstrap required, opened a new log
I20260809 04:39:35.252609 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:35.256407 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:35.254719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:43043 (local address 127.0.0.1:42275)
0809 04:39:35.254830 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:35.254833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:35.255102 (+ 269us) server_negotiation.cc:407] Connection header received
0809 04:39:35.255233 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:35.255236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:35.255269 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:35.255336 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:35.256042 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:35.256046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:35.256048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:35.256059 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:35.256087 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:35.256097 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:35.256163 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:35.256255 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 04:39:35.256289 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:39:35.256775 19396 raft_consensus.cc:348] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e19b102fe46c0a55235b6c82ebf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260809 04:39:35.257094 19396 raft_consensus.cc:374] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:35.257179 19396 raft_consensus.cc:729] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78e19b102fe46c0a55235b6c82ebf56, State: Initialized, Role: FOLLOWER
I20260809 04:39:35.257692 19396 consensus_queue.cc:260] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e19b102fe46c0a55235b6c82ebf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260809 04:39:35.257846 19396 raft_consensus.cc:388] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:35.257896 19396 raft_consensus.cc:482] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:35.257978 19396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:35.258908 19396 raft_consensus.cc:504] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e19b102fe46c0a55235b6c82ebf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260809 04:39:35.259233 19396 leader_election.cc:302] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a78e19b102fe46c0a55235b6c82ebf56; no voters:
I20260809 04:39:35.259523 19396 leader_election.cc:288] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:35.259676 19409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:35.259891 19409 raft_consensus.cc:686] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [term 1 LEADER]: Becoming Leader. State: Replica: a78e19b102fe46c0a55235b6c82ebf56, State: Running, Role: LEADER
I20260809 04:39:35.260250 19409 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e19b102fe46c0a55235b6c82ebf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260809 04:39:35.260797 19396 sys_catalog.cc:564] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:35.261863 19411 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78e19b102fe46c0a55235b6c82ebf56. Latest consensus state: current_term: 1 leader_uuid: "a78e19b102fe46c0a55235b6c82ebf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e19b102fe46c0a55235b6c82ebf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } } }
I20260809 04:39:35.262005 19411 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:35.262265 19410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a78e19b102fe46c0a55235b6c82ebf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e19b102fe46c0a55235b6c82ebf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } } }
I20260809 04:39:35.262382 19420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:35.262425 19410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:35.265730 19420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:35.272193 19420 catalog_manager.cc:1347] Generated new cluster ID: 122a49c4e337497cbe54218aaa36a610
I20260809 04:39:35.272269 19420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:35.285286 19420 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:35.285641 19420 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:35.295586 19420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a78e19b102fe46c0a55235b6c82ebf56: Generated new TSK 0
I20260809 04:39:35.296329 19420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:35.303751 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 19294
2026-08-09T04:39:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1499 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:35 dist-test-slave-rmhg krb5kdc[19443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:35 dist-test-slave-rmhg krb5kdc[19443](info): set up 2 sockets
Aug 09 04:39:35 dist-test-slave-rmhg krb5kdc[19443](info): commencing operation
krb5kdc: starting...
W20260809 04:39:36.447237 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:36 dist-test-slave-rmhg krb5kdc[19443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:36.572837 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:36.684420 19458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:36.684724 19458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:36.684789 19458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:36.688864 19458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:36.688941 19458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:36.688967 19458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:36.688989 19458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:36.689010 19458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:36.694419 19458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:36.695564 19458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:36.696615 19458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:39:36.705440 19458 server_base.cc:1034] running on GCE node
W20260809 04:39:36.705391 19472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:36.705401 19469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:36.705374 19470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:36.706207 19458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:36.707337 19458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:36.708500 19458 hybrid_clock.cc:648] HybridClock initialized: now 1786250376708480 us; error 33 us; skew 500 ppm
Aug 09 04:39:36 dist-test-slave-rmhg krb5kdc[19443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:36.711181 19458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:36.712069 19458 webserver.cc:460] Webserver started at http://127.0.0.1:39715/ using document root <none> and password file <none>
I20260809 04:39:36.712612 19458 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:36.712680 19458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:36.712874 19458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:36.715029 19458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "55a564127055420a81d91efed14c3034"
format_stamp: "Formatted at 2026-08-09 04:39:36 on dist-test-slave-rmhg"
server_key: "c58e58f547e096e7601d8346fb559a23"
server_key_iv: "8e9909cdcb5d7dcbcddff7c198aef4b4"
server_key_version: "encryptionkey@0"
I20260809 04:39:36.715576 19458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "55a564127055420a81d91efed14c3034"
format_stamp: "Formatted at 2026-08-09 04:39:36 on dist-test-slave-rmhg"
server_key: "c58e58f547e096e7601d8346fb559a23"
server_key_iv: "8e9909cdcb5d7dcbcddff7c198aef4b4"
server_key_version: "encryptionkey@0"
I20260809 04:39:36.719342 19458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:39:36.721824 19480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:36.723066 19458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:36.723183 19458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "55a564127055420a81d91efed14c3034"
format_stamp: "Formatted at 2026-08-09 04:39:36 on dist-test-slave-rmhg"
server_key: "c58e58f547e096e7601d8346fb559a23"
server_key_iv: "8e9909cdcb5d7dcbcddff7c198aef4b4"
server_key_version: "encryptionkey@0"
I20260809 04:39:36.723281 19458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:36.736752 19458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:36.739785 19458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:36.739987 19458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:36.747291 19458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37887
I20260809 04:39:36.747303 19555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37887 every 8 connection(s)
I20260809 04:39:36.748277 19458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:36.750116 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 19458
I20260809 04:39:36.750257 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:36.750590 19824 external_mini_cluster.cc:1428] Setting key efa472df6dcabccd4a37a96cd17fb009
I20260809 04:39:36.751221 19556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:36.758139 19556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034: Bootstrap starting.
Aug 09 04:39:36 dist-test-slave-rmhg krb5kdc[19443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 04:39:36.760617 19556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:36.761550 19556 log.cc:824] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:36.763425 19556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034: No bootstrap required, opened a new log
I20260809 04:39:36.765496 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:36.752047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:55243 (local address 127.0.0.1:37887)
0809 04:39:36.752462 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:36.752481 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:36.752523 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:39:36.753350 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:36.753375 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:36.753827 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:36.754214 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:36.754886 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:36.756366 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:36.757275 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:36.757949 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:36.758119 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:36.760737 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:36.760744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:36.760760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:36.760792 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:36.762542 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:36.763063 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:36.763068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:36.763073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:36.763189 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:36.763516 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:36.763520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:36.763522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:36.763955 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:36.764162 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:36.764503 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 04:39:36.764718 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260809 04:39:36.767208 19556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a564127055420a81d91efed14c3034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37887 } }
I20260809 04:39:36.767468 19556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:36.767580 19556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a564127055420a81d91efed14c3034, State: Initialized, Role: FOLLOWER
I20260809 04:39:36.768033 19556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a564127055420a81d91efed14c3034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37887 } }
I20260809 04:39:36.768187 19556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:36.768257 19556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:36.768359 19556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:36.768492 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:36.769300 19556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a564127055420a81d91efed14c3034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37887 } }
I20260809 04:39:36.769645 19556 leader_election.cc:302] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a564127055420a81d91efed14c3034; no voters:
I20260809 04:39:36.769982 19556 leader_election.cc:288] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:36.770076 19571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:36.770761 19571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [term 1 LEADER]: Becoming Leader. State: Replica: 55a564127055420a81d91efed14c3034, State: Running, Role: LEADER
I20260809 04:39:36.771122 19571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a564127055420a81d91efed14c3034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37887 } }
I20260809 04:39:36.771318 19556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:36.772392 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55a564127055420a81d91efed14c3034. Latest consensus state: current_term: 1 leader_uuid: "55a564127055420a81d91efed14c3034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a564127055420a81d91efed14c3034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37887 } } }
I20260809 04:39:36.772567 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:36.772782 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55a564127055420a81d91efed14c3034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a564127055420a81d91efed14c3034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37887 } } }
I20260809 04:39:36.772887 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:36.773015 19583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:36.776288 19583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:36.780221 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:36.770917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:36657 (local address 127.0.0.1:37887)
0809 04:39:36.771025 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:36.771029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:36.771040 (+ 11us) server_negotiation.cc:407] Connection header received
0809 04:39:36.771069 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:36.771071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:36.771106 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:36.771172 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:36.772032 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:36.772838 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:36.774076 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:36.774426 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:36.775455 (+ 1029us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:36.776256 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:36.776259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:36.776262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:36.776287 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:36.778187 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:36.778781 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:36.778784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:36.778786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:36.778878 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:36.779249 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:36.779253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:36.779254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:36.779472 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:36.779632 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:36.779972 (+ 340us) server_negotiation.cc:299] Negotiation successful
0809 04:39:36.780074 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 04:39:36.782807 19583 catalog_manager.cc:1347] Generated new cluster ID: fcd56912c8f847f2b684624b9f3010a0
I20260809 04:39:36.782882 19583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:36.798980 19583 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:36.800149 19583 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:36.806375 19583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55a564127055420a81d91efed14c3034: Generated new TSK 0
I20260809 04:39:36.807070 19583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:36.811144 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 19458
W20260809 04:39:36.825435 19438 connection.cc:535] client connection to 127.0.0.1:37887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37887): Connection reset by peer (error 104)
2026-08-09T04:39:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:36 dist-test-slave-rmhg krb5kdc[19597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:36 dist-test-slave-rmhg krb5kdc[19597](info): set up 2 sockets
Aug 09 04:39:36 dist-test-slave-rmhg krb5kdc[19597](info): commencing operation
krb5kdc: starting...
W20260809 04:39:37.950352 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:38 dist-test-slave-rmhg krb5kdc[19597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:38.072398 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:38.183773 19618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:38.184070 19618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:38.184115 19618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:38.188292 19618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:38.188360 19618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:38.188380 19618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:38.188396 19618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:38.188410 19618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:38.194541 19618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:38.195627 19618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:38.196614 19618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:38.205132 19624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:38.205317 19623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:38.205137 19627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:38.205228 19618 server_base.cc:1034] running on GCE node
I20260809 04:39:38.206060 19618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:38.207163 19618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:38.208323 19618 hybrid_clock.cc:648] HybridClock initialized: now 1786250378208303 us; error 34 us; skew 500 ppm
Aug 09 04:39:38 dist-test-slave-rmhg krb5kdc[19597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:38.211059 19618 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:38.211962 19618 webserver.cc:460] Webserver started at http://127.0.0.1:38195/ using document root <none> and password file <none>
I20260809 04:39:38.212512 19618 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:38.212579 19618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:38.212787 19618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:38.214874 19618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "fc6364af296e41fc93a2eda320a8ad99"
format_stamp: "Formatted at 2026-08-09 04:39:38 on dist-test-slave-rmhg"
server_key: "5d63a059ffe56612caf9f65ff0d6b058"
server_key_iv: "232c83111cf640f179f48368f867fef8"
server_key_version: "encryptionkey@0"
I20260809 04:39:38.215430 19618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "fc6364af296e41fc93a2eda320a8ad99"
format_stamp: "Formatted at 2026-08-09 04:39:38 on dist-test-slave-rmhg"
server_key: "5d63a059ffe56612caf9f65ff0d6b058"
server_key_iv: "232c83111cf640f179f48368f867fef8"
server_key_version: "encryptionkey@0"
I20260809 04:39:38.219205 19618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 04:39:38.221673 19637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:38.222968 19618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:38.223100 19618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "fc6364af296e41fc93a2eda320a8ad99"
format_stamp: "Formatted at 2026-08-09 04:39:38 on dist-test-slave-rmhg"
server_key: "5d63a059ffe56612caf9f65ff0d6b058"
server_key_iv: "232c83111cf640f179f48368f867fef8"
server_key_version: "encryptionkey@0"
I20260809 04:39:38.223197 19618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:38.247079 19618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:38.250312 19618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:38.250506 19618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:38.257463 19618 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36803
I20260809 04:39:38.257491 19717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36803 every 8 connection(s)
I20260809 04:39:38.258433 19618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:38.259061 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 19618
I20260809 04:39:38.259200 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:38.259485 19824 external_mini_cluster.cc:1428] Setting key 77498a73d5cf4c38e0d3dc75dafc9a72
I20260809 04:39:38.261603 19718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:38.267683 19718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99: Bootstrap starting.
I20260809 04:39:38.268857 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:38.260902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:51773 (local address 127.0.0.1:36803)
0809 04:39:38.261401 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:38.261416 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:38.261457 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:39:38.262157 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:38.262176 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:38.262533 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:38.262858 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:38.263292 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:38.264294 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:38.265238 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:38.265934 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:38.266106 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:38.266934 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:38.266941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:38.266957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:38.267447 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:38.267513 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:38.267544 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:38.267676 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:38.267810 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 04:39:38.267948 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":167,"threads_started":1}
I20260809 04:39:38.270376 19718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:38.271209 19718 log.cc:824] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:38.271925 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:38.273340 19718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99: No bootstrap required, opened a new log
I20260809 04:39:38.277114 19718 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6364af296e41fc93a2eda320a8ad99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36803 } }
I20260809 04:39:38.277388 19718 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:38.277498 19718 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6364af296e41fc93a2eda320a8ad99, State: Initialized, Role: FOLLOWER
I20260809 04:39:38.277676 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:38.273902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:45871 (local address 127.0.0.1:36803)
0809 04:39:38.274007 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:38.274012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:38.274210 (+ 198us) server_negotiation.cc:407] Connection header received
0809 04:39:38.274326 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:38.274329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:38.274371 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:38.274434 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:38.274727 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:38.275244 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:38.276045 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:38.276278 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:38.276365 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:38.277139 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:38.277142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:38.277145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:38.277182 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:38.277212 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:38.277224 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:38.277302 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:38.277498 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 04:39:38.277551 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 04:39:38.278031 19718 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6364af296e41fc93a2eda320a8ad99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36803 } }
I20260809 04:39:38.278199 19718 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:38.278268 19718 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:38.278367 19718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:38.279371 19718 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6364af296e41fc93a2eda320a8ad99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36803 } }
I20260809 04:39:38.279709 19718 leader_election.cc:302] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc6364af296e41fc93a2eda320a8ad99; no voters:
I20260809 04:39:38.280000 19718 leader_election.cc:288] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:38.280217 19733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:38.280437 19733 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [term 1 LEADER]: Becoming Leader. State: Replica: fc6364af296e41fc93a2eda320a8ad99, State: Running, Role: LEADER
I20260809 04:39:38.280790 19733 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6364af296e41fc93a2eda320a8ad99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36803 } }
I20260809 04:39:38.281231 19718 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:38.282374 19735 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc6364af296e41fc93a2eda320a8ad99. Latest consensus state: current_term: 1 leader_uuid: "fc6364af296e41fc93a2eda320a8ad99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6364af296e41fc93a2eda320a8ad99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36803 } } }
I20260809 04:39:38.282513 19735 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:38.282846 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc6364af296e41fc93a2eda320a8ad99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6364af296e41fc93a2eda320a8ad99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36803 } } }
I20260809 04:39:38.282940 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:38.284278 19748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:38.287277 19748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:38.293205 19748 catalog_manager.cc:1347] Generated new cluster ID: 17ea9214fdb24691aebb83c130e73639
I20260809 04:39:38.293277 19748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:38.307986 19748 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:38.309612 19748 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:38.317710 19748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc6364af296e41fc93a2eda320a8ad99: Generated new TSK 0
I20260809 04:39:38.318506 19748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:38.325110 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 19618
W20260809 04:39:38.340287 19594 connection.cc:535] client connection to 127.0.0.1:36803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36803): Connection reset by peer (error 104)
2026-08-09T04:39:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:38 dist-test-slave-rmhg krb5kdc[19762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:38 dist-test-slave-rmhg krb5kdc[19762](info): set up 2 sockets
Aug 09 04:39:38 dist-test-slave-rmhg krb5kdc[19762](info): commencing operation
krb5kdc: starting...
W20260809 04:39:39.458026 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:39 dist-test-slave-rmhg krb5kdc[19762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:39:39.586391 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:39.701579 19780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:39.701867 19780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:39.701925 19780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:39.705935 19780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:39.706012 19780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:39.706038 19780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:39.706069 19780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:39.706099 19780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:39.711416 19780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:39.712476 19780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:39.713560 19780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:39:39.722043 19780 server_base.cc:1034] running on GCE node
W20260809 04:39:39.721959 19788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:39.721961 19787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:39.721956 19790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:39.722895 19780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:39.724066 19780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:39.725234 19780 hybrid_clock.cc:648] HybridClock initialized: now 1786250379725207 us; error 37 us; skew 500 ppm
Aug 09 04:39:39 dist-test-slave-rmhg krb5kdc[19762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:39.727934 19780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:39:39.728811 19780 webserver.cc:460] Webserver started at http://127.0.0.1:43679/ using document root <none> and password file <none>
I20260809 04:39:39.729405 19780 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:39.729481 19780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:39.729663 19780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:39.731690 19780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "812e8837b0e843e58b58f3eb6dba135b"
format_stamp: "Formatted at 2026-08-09 04:39:39 on dist-test-slave-rmhg"
server_key: "d754e135cddd1aa0830ef1b30693b66d"
server_key_iv: "5f8f7e58a372a856fb7522f71fdc367e"
server_key_version: "encryptionkey@0"
I20260809 04:39:39.732208 19780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "812e8837b0e843e58b58f3eb6dba135b"
format_stamp: "Formatted at 2026-08-09 04:39:39 on dist-test-slave-rmhg"
server_key: "d754e135cddd1aa0830ef1b30693b66d"
server_key_iv: "5f8f7e58a372a856fb7522f71fdc367e"
server_key_version: "encryptionkey@0"
I20260809 04:39:39.735860 19780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 04:39:39.738273 19800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:39.739566 19780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:39:39.739674 19780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "812e8837b0e843e58b58f3eb6dba135b"
format_stamp: "Formatted at 2026-08-09 04:39:39 on dist-test-slave-rmhg"
server_key: "d754e135cddd1aa0830ef1b30693b66d"
server_key_iv: "5f8f7e58a372a856fb7522f71fdc367e"
server_key_version: "encryptionkey@0"
I20260809 04:39:39.739765 19780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:39.744388 19780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:39.747215 19780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:39.747363 19780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:39.754063 19780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34327
I20260809 04:39:39.754092 19876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34327 every 8 connection(s)
I20260809 04:39:39.755048 19780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:39.758023 19878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:39.763900 19878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b: Bootstrap starting.
I20260809 04:39:39.764204 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 19780
I20260809 04:39:39.764353 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:39.764668 19824 external_mini_cluster.cc:1428] Setting key fd7ecb1fe7f7308aa924db992cb99c47
I20260809 04:39:39.766799 19878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:39.767529 19878 log.cc:824] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:39.769588 19878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b: No bootstrap required, opened a new log
I20260809 04:39:39.769985 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:39.766067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:44875 (local address 127.0.0.1:34327)
0809 04:39:39.766576 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:39.766592 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:39.766630 (+ 38us) server_negotiation.cc:407] Connection header received
0809 04:39:39.767328 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:39.767350 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:39.767672 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:39.768020 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:39.768717 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:39.768722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:39.768734 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:39.768745 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:39.768801 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:39.768820 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:39.768916 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:39.769045 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 04:39:39.769237 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":144,"threads_started":1}
I20260809 04:39:39.772961 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:39.773571 19878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812e8837b0e843e58b58f3eb6dba135b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260809 04:39:39.773782 19878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:39.773818 19878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812e8837b0e843e58b58f3eb6dba135b, State: Initialized, Role: FOLLOWER
I20260809 04:39:39.774276 19878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812e8837b0e843e58b58f3eb6dba135b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260809 04:39:39.774411 19878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:39.774472 19878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:39.774569 19878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:39.775655 19878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812e8837b0e843e58b58f3eb6dba135b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260809 04:39:39.776033 19878 leader_election.cc:302] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 812e8837b0e843e58b58f3eb6dba135b; no voters:
I20260809 04:39:39.776397 19878 leader_election.cc:288] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:39.776526 19893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:39.776657 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:39.775020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:38937 (local address 127.0.0.1:34327)
0809 04:39:39.775134 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:39.775139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:39.775276 (+ 137us) server_negotiation.cc:407] Connection header received
0809 04:39:39.775388 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:39.775390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:39.775420 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:39.775482 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:39.776209 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:39.776213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:39.776216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:39.776228 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:39.776258 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:39.776269 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:39.776337 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:39.776454 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 04:39:39.776494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:39:39.776834 19893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [term 1 LEADER]: Becoming Leader. State: Replica: 812e8837b0e843e58b58f3eb6dba135b, State: Running, Role: LEADER
I20260809 04:39:39.777211 19893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812e8837b0e843e58b58f3eb6dba135b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260809 04:39:39.777802 19878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:39.778419 19895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 812e8837b0e843e58b58f3eb6dba135b. Latest consensus state: current_term: 1 leader_uuid: "812e8837b0e843e58b58f3eb6dba135b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812e8837b0e843e58b58f3eb6dba135b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } } }
I20260809 04:39:39.778589 19895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:39.778872 19894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "812e8837b0e843e58b58f3eb6dba135b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812e8837b0e843e58b58f3eb6dba135b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } } }
I20260809 04:39:39.778982 19894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b [sys.catalog]: This master's current role is: LEADER
W20260809 04:39:39.782815 19913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:39.782889 19913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:39.782966 19905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:39.783664 19905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:39.789532 19905 catalog_manager.cc:1347] Generated new cluster ID: bbf48339d2ed4159861e60a337ef6787
I20260809 04:39:39.789592 19905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:39.803925 19905 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:39.804266 19905 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:39.812685 19905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 812e8837b0e843e58b58f3eb6dba135b: Generated new TSK 0
I20260809 04:39:39.813655 19905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:39.818758 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 19780
2026-08-09T04:39:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:39 dist-test-slave-rmhg krb5kdc[19921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:39 dist-test-slave-rmhg krb5kdc[19921](info): set up 2 sockets
Aug 09 04:39:39 dist-test-slave-rmhg krb5kdc[19921](info): commencing operation
krb5kdc: starting...
W20260809 04:39:40.952057 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:41 dist-test-slave-rmhg krb5kdc[19921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:41.077782 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41159
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:41.191478 19939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:41.191766 19939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:41.191809 19939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:41.195890 19939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:41.195959 19939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:41.195978 19939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:41.196007 19939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:41.196022 19939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:41.201794 19939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:41.203310 19939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:41.204351 19939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:41.212877 19946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:41.212949 19939 server_base.cc:1034] running on GCE node
W20260809 04:39:41.212839 19950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:41.212852 19948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:41.213749 19939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:41.214895 19939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:41.216066 19939 hybrid_clock.cc:648] HybridClock initialized: now 1786250381216047 us; error 30 us; skew 500 ppm
Aug 09 04:39:41 dist-test-slave-rmhg krb5kdc[19921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250381, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:41.223387 19939 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:41.224475 19939 webserver.cc:460] Webserver started at http://127.19.92.62:36387/ using document root <none> and password file <none>
I20260809 04:39:41.225082 19939 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:41.225173 19939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:41.225414 19939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:41.227617 19939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "9102566f6d554d5b8d264c3ee0c44174"
format_stamp: "Formatted at 2026-08-09 04:39:41 on dist-test-slave-rmhg"
server_key: "3cd4354fbfbb6e36025f0e55de47d149"
server_key_iv: "66e2744c03724238cc4facdeadd0aed7"
server_key_version: "encryptionkey@0"
I20260809 04:39:41.228217 19939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "9102566f6d554d5b8d264c3ee0c44174"
format_stamp: "Formatted at 2026-08-09 04:39:41 on dist-test-slave-rmhg"
server_key: "3cd4354fbfbb6e36025f0e55de47d149"
server_key_iv: "66e2744c03724238cc4facdeadd0aed7"
server_key_version: "encryptionkey@0"
I20260809 04:39:41.232239 19939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260809 04:39:41.234865 19960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:41.236207 19939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:39:41.236351 19939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "9102566f6d554d5b8d264c3ee0c44174"
format_stamp: "Formatted at 2026-08-09 04:39:41 on dist-test-slave-rmhg"
server_key: "3cd4354fbfbb6e36025f0e55de47d149"
server_key_iv: "66e2744c03724238cc4facdeadd0aed7"
server_key_version: "encryptionkey@0"
I20260809 04:39:41.236474 19939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:41.248719 19939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:41.251669 19939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:41.251852 19939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:41.258872 19939 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41159
I20260809 04:39:41.258893 20034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41159 every 8 connection(s)
I20260809 04:39:41.259897 19939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:41.262773 20035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:41.264856 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 19939
I20260809 04:39:41.265017 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:41.265321 19824 external_mini_cluster.cc:1428] Setting key 16fe1f659591441c2875247ff46dfb63
I20260809 04:39:41.269133 20035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174: Bootstrap starting.
I20260809 04:39:41.271776 20035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:41.272656 20035 log.cc:824] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:41.274645 20035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174: No bootstrap required, opened a new log
Aug 09 04:39:41 dist-test-slave-rmhg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:41.278508 20035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9102566f6d554d5b8d264c3ee0c44174" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41159 } }
I20260809 04:39:41.278797 20035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:41.278899 20035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9102566f6d554d5b8d264c3ee0c44174, State: Initialized, Role: FOLLOWER
I20260809 04:39:41.279450 20035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9102566f6d554d5b8d264c3ee0c44174" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41159 } }
I20260809 04:39:41.279627 20035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:41.279702 20035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:41.279806 20035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:41.280798 20035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9102566f6d554d5b8d264c3ee0c44174" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41159 } }
I20260809 04:39:41.281050 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:41.266740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58469 (local address 127.19.92.62:41159)
0809 04:39:41.267163 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:41.267189 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:41.267233 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:39:41.268028 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:41.268053 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:41.268493 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:41.268857 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:41.269627 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:41.271108 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:41.272030 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:41.272731 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:41.272899 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:41.275393 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:41.275400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:41.275436 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:41.275468 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:41.277897 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:41.278396 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:41.278401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:41.278418 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:41.278568 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:41.278892 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:41.278896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:41.278898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:41.279456 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:41.279689 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:41.280018 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 04:39:41.280241 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":106,"threads_started":1}
I20260809 04:39:41.281191 20035 leader_election.cc:302] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9102566f6d554d5b8d264c3ee0c44174; no voters:
I20260809 04:39:41.281549 20035 leader_election.cc:288] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:41.281626 20041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:41.281823 20041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [term 1 LEADER]: Becoming Leader. State: Replica: 9102566f6d554d5b8d264c3ee0c44174, State: Running, Role: LEADER
I20260809 04:39:41.282210 20041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9102566f6d554d5b8d264c3ee0c44174" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41159 } }
I20260809 04:39:41.282930 20035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:41.284003 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9102566f6d554d5b8d264c3ee0c44174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9102566f6d554d5b8d264c3ee0c44174" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41159 } } }
I20260809 04:39:41.284307 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:41.284399 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:41.284287 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9102566f6d554d5b8d264c3ee0c44174. Latest consensus state: current_term: 1 leader_uuid: "9102566f6d554d5b8d264c3ee0c44174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9102566f6d554d5b8d264c3ee0c44174" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41159 } } }
I20260809 04:39:41.284615 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:41.284888 20063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:41.287963 20063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:41.294004 20063 catalog_manager.cc:1347] Generated new cluster ID: 24bf51bae14447bca26a0d843d174b0e
I20260809 04:39:41.294296 20063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:41.296527 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:41.286497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44397 (local address 127.19.92.62:41159)
0809 04:39:41.286599 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:41.286603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:41.288322 (+ 1719us) server_negotiation.cc:407] Connection header received
0809 04:39:41.288434 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:41.288437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:41.288489 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:41.288573 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:41.289396 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:41.290145 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:41.291278 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:41.291639 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:41.291748 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:41.293083 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:41.293086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:41.293088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:41.293110 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:41.294747 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:41.295278 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:41.295282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:41.295284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:41.295385 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:41.295673 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:41.295676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:41.295677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:41.295889 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:41.296056 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:41.296269 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 04:39:41.296374 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 04:39:41.317054 20063 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:41.318434 20063 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:41.326189 20063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9102566f6d554d5b8d264c3ee0c44174: Generated new TSK 0
I20260809 04:39:41.327179 20063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:41.332532 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 19939
W20260809 04:39:41.347517 19918 connection.cc:535] client connection to 127.19.92.62:41159 recv error: Network error: failed to read from TLS socket (remote: 127.19.92.62:41159): Connection reset by peer (error 104)
2026-08-09T04:39:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:41 dist-test-slave-rmhg krb5kdc[20085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:41 dist-test-slave-rmhg krb5kdc[20085](info): set up 2 sockets
Aug 09 04:39:41 dist-test-slave-rmhg krb5kdc[20085](info): commencing operation
krb5kdc: starting...
W20260809 04:39:42.467115 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:42 dist-test-slave-rmhg krb5kdc[20085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:42.596354 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:40321
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:44035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:40321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:42.712605 20104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:42.712872 20104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:42.712939 20104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:42.716866 20104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:42.716929 20104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:42.716955 20104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:42.716975 20104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:42.716995 20104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:42.722111 20104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:44035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:40321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:42.723133 20104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:42.724144 20104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:42.732353 20115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:42.732393 20110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:42.732563 20104 server_base.cc:1034] running on GCE node
W20260809 04:39:42.732389 20113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:42.733331 20104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:42.734391 20104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:42.735562 20104 hybrid_clock.cc:648] HybridClock initialized: now 1786250382735543 us; error 32 us; skew 500 ppm
Aug 09 04:39:42 dist-test-slave-rmhg krb5kdc[20085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250382, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:42.738381 20104 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:42.739261 20104 webserver.cc:460] Webserver started at http://127.19.92.62:43519/ using document root <none> and password file <none>
I20260809 04:39:42.739791 20104 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:42.739857 20104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:42.740078 20104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:42.742362 20104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "594c6d46585e49928d0d1e60b54557d8"
format_stamp: "Formatted at 2026-08-09 04:39:42 on dist-test-slave-rmhg"
server_key: "3a66a4c4fac0fadedb3de24b1a71c04c"
server_key_iv: "bd6cc062f1f7451fe3b74ea291d35840"
server_key_version: "encryptionkey@0"
I20260809 04:39:42.742923 20104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "594c6d46585e49928d0d1e60b54557d8"
format_stamp: "Formatted at 2026-08-09 04:39:42 on dist-test-slave-rmhg"
server_key: "3a66a4c4fac0fadedb3de24b1a71c04c"
server_key_iv: "bd6cc062f1f7451fe3b74ea291d35840"
server_key_version: "encryptionkey@0"
I20260809 04:39:42.746603 20104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:39:42.749004 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:42.750226 20104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:42.750339 20104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "594c6d46585e49928d0d1e60b54557d8"
format_stamp: "Formatted at 2026-08-09 04:39:42 on dist-test-slave-rmhg"
server_key: "3a66a4c4fac0fadedb3de24b1a71c04c"
server_key_iv: "bd6cc062f1f7451fe3b74ea291d35840"
server_key_version: "encryptionkey@0"
I20260809 04:39:42.750427 20104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:42.771588 20104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:42.774850 20104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:42.775036 20104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:42.782505 20104 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:40321
I20260809 04:39:42.782588 20199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:40321 every 8 connection(s)
I20260809 04:39:42.783529 20104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:42.784098 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20104
I20260809 04:39:42.784250 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:42.784546 19824 external_mini_cluster.cc:1428] Setting key 104c8eeed0ead0f4f117c861305bea66
I20260809 04:39:42.786885 20202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:42.792884 20202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8: Bootstrap starting.
I20260809 04:39:42.794168 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:42.786024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46025 (local address 127.19.92.62:40321)
0809 04:39:42.786485 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:42.786502 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:42.786545 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:39:42.787222 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:42.787242 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:42.787587 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:42.787898 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:42.788330 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:42.789386 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:42.790299 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:42.790781 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:42.790927 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:42.791897 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:42.791902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:42.791914 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:42.792307 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:42.792363 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:42.792387 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:42.792503 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:42.792729 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 04:39:42.793087 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":146,"threads_started":1}
I20260809 04:39:42.795360 20202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:42.796218 20202 log.cc:824] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:42.797312 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:42.798286 20202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8: No bootstrap required, opened a new log
I20260809 04:39:42.801975 20202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594c6d46585e49928d0d1e60b54557d8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40321 } }
I20260809 04:39:42.802244 20202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:42.802307 20202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594c6d46585e49928d0d1e60b54557d8, State: Initialized, Role: FOLLOWER
I20260809 04:39:42.802769 20202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594c6d46585e49928d0d1e60b54557d8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40321 } }
I20260809 04:39:42.802911 20202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:42.802977 20202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:42.803093 20202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:42.803133 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:42.799256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58821 (local address 127.19.92.62:40321)
0809 04:39:42.799365 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:42.799369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:42.799579 (+ 210us) server_negotiation.cc:407] Connection header received
0809 04:39:42.799667 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:42.799670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:42.799710 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:42.799771 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:42.800052 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:42.800572 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:42.801457 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:42.801696 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:42.801786 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:42.802645 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:42.802648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:42.802650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:42.802661 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:42.802685 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:42.802696 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:42.802767 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:42.802915 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 04:39:42.802952 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 04:39:42.804201 20202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594c6d46585e49928d0d1e60b54557d8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40321 } }
I20260809 04:39:42.804539 20202 leader_election.cc:302] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594c6d46585e49928d0d1e60b54557d8; no voters:
I20260809 04:39:42.804831 20202 leader_election.cc:288] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:42.804934 20213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:42.805181 20213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [term 1 LEADER]: Becoming Leader. State: Replica: 594c6d46585e49928d0d1e60b54557d8, State: Running, Role: LEADER
I20260809 04:39:42.805464 20213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594c6d46585e49928d0d1e60b54557d8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40321 } }
I20260809 04:39:42.806012 20202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:42.806985 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594c6d46585e49928d0d1e60b54557d8. Latest consensus state: current_term: 1 leader_uuid: "594c6d46585e49928d0d1e60b54557d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594c6d46585e49928d0d1e60b54557d8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40321 } } }
I20260809 04:39:42.807112 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:42.807425 20215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "594c6d46585e49928d0d1e60b54557d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594c6d46585e49928d0d1e60b54557d8" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40321 } } }
I20260809 04:39:42.807520 20215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:42.807581 20225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:42.810806 20225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:42.816381 20225 catalog_manager.cc:1347] Generated new cluster ID: 49241da646fb4b3db33dcd638653204f
I20260809 04:39:42.816454 20225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:42.838240 20225 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:42.839279 20225 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:42.843905 20225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 594c6d46585e49928d0d1e60b54557d8: Generated new TSK 0
I20260809 04:39:42.844611 20225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:42.849648 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20104
W20260809 04:39:42.859246 20083 connection.cc:535] client connection to 127.19.92.62:40321 recv error: Network error: failed to read from TLS socket (remote: 127.19.92.62:40321): Connection reset by peer (error 104)
2026-08-09T04:39:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:42 dist-test-slave-rmhg krb5kdc[20241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:42 dist-test-slave-rmhg krb5kdc[20241](info): set up 2 sockets
Aug 09 04:39:42 dist-test-slave-rmhg krb5kdc[20241](info): commencing operation
krb5kdc: starting...
W20260809 04:39:43.971606 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:44 dist-test-slave-rmhg krb5kdc[20241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:44.100713 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:40953
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:40953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:44.213614 20259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:44.213860 20259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:44.213900 20259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:44.217839 20259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:44.217896 20259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:44.217914 20259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:44.217929 20259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:44.217943 20259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:44.223336 20259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:40953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:40953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:44.224340 20259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:44.225443 20259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:44.233755 20267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:44.233714 20266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:44.233775 20259 server_base.cc:1034] running on GCE node
W20260809 04:39:44.233727 20269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:44.234496 20259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:44.235608 20259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:44.236779 20259 hybrid_clock.cc:648] HybridClock initialized: now 1786250384236764 us; error 26 us; skew 500 ppm
Aug 09 04:39:44 dist-test-slave-rmhg krb5kdc[20241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250384, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:44.239475 20259 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:44.240348 20259 webserver.cc:460] Webserver started at http://127.19.92.62:42123/ using document root <none> and password file <none>
I20260809 04:39:44.240875 20259 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:44.240933 20259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:44.241106 20259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:44.243172 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "bf78176cffc84c579da4263209db65b9"
format_stamp: "Formatted at 2026-08-09 04:39:44 on dist-test-slave-rmhg"
server_key: "736e744fa03ef1704950bc2207a875ac"
server_key_iv: "cf4c8010e83ff6a4a71d434171052294"
server_key_version: "encryptionkey@0"
I20260809 04:39:44.243731 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "bf78176cffc84c579da4263209db65b9"
format_stamp: "Formatted at 2026-08-09 04:39:44 on dist-test-slave-rmhg"
server_key: "736e744fa03ef1704950bc2207a875ac"
server_key_iv: "cf4c8010e83ff6a4a71d434171052294"
server_key_version: "encryptionkey@0"
I20260809 04:39:44.247512 20259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 04:39:44.249930 20279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:44.251150 20259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:44.251257 20259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "bf78176cffc84c579da4263209db65b9"
format_stamp: "Formatted at 2026-08-09 04:39:44 on dist-test-slave-rmhg"
server_key: "736e744fa03ef1704950bc2207a875ac"
server_key_iv: "cf4c8010e83ff6a4a71d434171052294"
server_key_version: "encryptionkey@0"
I20260809 04:39:44.251363 20259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:44.255992 20259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:44.259060 20259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:44.259231 20259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:44.266202 20259 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:40953
I20260809 04:39:44.266216 20357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:40953 every 8 connection(s)
I20260809 04:39:44.267170 20259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:44.267601 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20259
I20260809 04:39:44.267748 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:44.268034 19824 external_mini_cluster.cc:1428] Setting key 59445e658a14db5a637a96082d825f86
I20260809 04:39:44.270395 20358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:44.273494 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:44.269544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56903 (local address 127.19.92.62:40953)
0809 04:39:44.270022 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:44.270038 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:44.270075 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:39:44.270749 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:44.270768 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:44.271074 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:44.271406 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:44.272214 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:44.272219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:44.272230 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:44.272240 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:44.272297 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:44.272316 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:44.272445 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:44.272544 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 04:39:44.272700 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":147,"threads_started":1}
I20260809 04:39:44.276437 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:44.276633 20358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9: Bootstrap starting.
I20260809 04:39:44.278899 20358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:44.279685 20358 log.cc:824] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:44.280128 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:44.278506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49923 (local address 127.19.92.62:40953)
0809 04:39:44.278637 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:44.278641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:44.278863 (+ 222us) server_negotiation.cc:407] Connection header received
0809 04:39:44.278984 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:44.278987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:44.279024 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:44.279091 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:44.279771 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:44.279775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:44.279778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:44.279789 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:44.279818 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:44.279831 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:44.279911 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:44.279978 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 04:39:44.280016 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 04:39:44.281831 20358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9: No bootstrap required, opened a new log
I20260809 04:39:44.285492 20358 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf78176cffc84c579da4263209db65b9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40953 } }
I20260809 04:39:44.285730 20358 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:44.285772 20358 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf78176cffc84c579da4263209db65b9, State: Initialized, Role: FOLLOWER
I20260809 04:39:44.286155 20358 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf78176cffc84c579da4263209db65b9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40953 } }
I20260809 04:39:44.286280 20358 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:44.286326 20358 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:44.286396 20358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:44.287290 20358 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf78176cffc84c579da4263209db65b9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40953 } }
I20260809 04:39:44.287644 20358 leader_election.cc:302] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf78176cffc84c579da4263209db65b9; no voters:
I20260809 04:39:44.287989 20358 leader_election.cc:288] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:44.288115 20368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:44.288297 20368 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [term 1 LEADER]: Becoming Leader. State: Replica: bf78176cffc84c579da4263209db65b9, State: Running, Role: LEADER
I20260809 04:39:44.288647 20368 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf78176cffc84c579da4263209db65b9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40953 } }
I20260809 04:39:44.289289 20358 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:44.290270 20370 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf78176cffc84c579da4263209db65b9. Latest consensus state: current_term: 1 leader_uuid: "bf78176cffc84c579da4263209db65b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf78176cffc84c579da4263209db65b9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40953 } } }
I20260809 04:39:44.290396 20370 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:44.290700 20369 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf78176cffc84c579da4263209db65b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf78176cffc84c579da4263209db65b9" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 40953 } } }
I20260809 04:39:44.290818 20369 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:44.290903 20382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:44.294139 20382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:44.300005 20382 catalog_manager.cc:1347] Generated new cluster ID: 9ed5e232ceee494ba210a512829ec3de
I20260809 04:39:44.300066 20382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:44.314479 20382 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:44.314812 20382 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:44.321957 20382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf78176cffc84c579da4263209db65b9: Generated new TSK 0
I20260809 04:39:44.322666 20382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:44.330013 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20259
2026-08-09T04:39:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:44 dist-test-slave-rmhg krb5kdc[20399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:44 dist-test-slave-rmhg krb5kdc[20399](info): set up 2 sockets
Aug 09 04:39:44 dist-test-slave-rmhg krb5kdc[20399](info): commencing operation
krb5kdc: starting...
W20260809 04:39:45.470928 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:45 dist-test-slave-rmhg krb5kdc[20399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:45.598165 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:33943
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:32877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:45.707527 20418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:45.707818 20418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:45.707878 20418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:45.711918 20418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:45.711992 20418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:45.712019 20418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:45.712041 20418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:45.712060 20418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:45.717437 20418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:32877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:33943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:45.718528 20418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:45.719590 20418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:45.728317 20429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:45.728354 20418 server_base.cc:1034] running on GCE node
W20260809 04:39:45.728323 20424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:45.728322 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:45.729223 20418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:45.730340 20418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:45.731506 20418 hybrid_clock.cc:648] HybridClock initialized: now 1786250385731488 us; error 33 us; skew 500 ppm
Aug 09 04:39:45 dist-test-slave-rmhg krb5kdc[20399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250385, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:45.734257 20418 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:45.735172 20418 webserver.cc:460] Webserver started at http://127.19.92.62:42713/ using document root <none> and password file <none>
I20260809 04:39:45.735695 20418 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:45.735754 20418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:45.735917 20418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:45.738356 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "fa3f65825cc5467fac17da0d6931a0cd"
format_stamp: "Formatted at 2026-08-09 04:39:45 on dist-test-slave-rmhg"
server_key: "a02e9367425f6024ec84b657be8fc02c"
server_key_iv: "968fb4da33f064826e59d12b049e1503"
server_key_version: "encryptionkey@0"
I20260809 04:39:45.738878 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "fa3f65825cc5467fac17da0d6931a0cd"
format_stamp: "Formatted at 2026-08-09 04:39:45 on dist-test-slave-rmhg"
server_key: "a02e9367425f6024ec84b657be8fc02c"
server_key_iv: "968fb4da33f064826e59d12b049e1503"
server_key_version: "encryptionkey@0"
I20260809 04:39:45.742586 20418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:39:45.744901 20442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:45.746181 20418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:45.746290 20418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "fa3f65825cc5467fac17da0d6931a0cd"
format_stamp: "Formatted at 2026-08-09 04:39:45 on dist-test-slave-rmhg"
server_key: "a02e9367425f6024ec84b657be8fc02c"
server_key_iv: "968fb4da33f064826e59d12b049e1503"
server_key_version: "encryptionkey@0"
I20260809 04:39:45.746374 20418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:45.761117 20418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:45.764249 20418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:45.764436 20418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:45.771503 20418 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:33943
I20260809 04:39:45.771570 20518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:33943 every 8 connection(s)
I20260809 04:39:45.772509 20418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:45.774833 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20418
I20260809 04:39:45.774963 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:45.775240 19824 external_mini_cluster.cc:1428] Setting key 8a04b94d68754a0ec6ae9c7d94a5ea06
I20260809 04:39:45.775521 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:45.782917 20519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd: Bootstrap starting.
Aug 09 04:39:45 dist-test-slave-rmhg krb5kdc[20399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:45.785333 20519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:45.786281 20519 log.cc:824] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:45.788650 20519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd: No bootstrap required, opened a new log
I20260809 04:39:45.790915 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:45.776704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60565 (local address 127.19.92.62:33943)
0809 04:39:45.777170 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:45.777193 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:45.777237 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:39:45.777977 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:45.778003 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:45.778479 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:45.778876 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:45.779617 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:45.781112 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:45.782064 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:45.782776 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:45.782939 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:45.785431 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:45.785438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:45.785456 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:45.785493 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:45.787395 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:45.787897 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:45.787903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:45.787909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:45.788050 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:45.788339 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:45.788342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:45.788344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:45.788884 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:45.789076 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:45.789385 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 04:39:45.789705 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260809 04:39:45.792434 20519 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3f65825cc5467fac17da0d6931a0cd" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33943 } }
I20260809 04:39:45.792668 20519 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:45.792775 20519 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3f65825cc5467fac17da0d6931a0cd, State: Initialized, Role: FOLLOWER
I20260809 04:39:45.793292 20519 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3f65825cc5467fac17da0d6931a0cd" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33943 } }
I20260809 04:39:45.793480 20519 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:45.793550 20519 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:45.793653 20519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:45.793874 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:45.794828 20519 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3f65825cc5467fac17da0d6931a0cd" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33943 } }
I20260809 04:39:45.795301 20519 leader_election.cc:302] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa3f65825cc5467fac17da0d6931a0cd; no voters:
I20260809 04:39:45.795681 20519 leader_election.cc:288] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:45.795765 20534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:45.795998 20534 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [term 1 LEADER]: Becoming Leader. State: Replica: fa3f65825cc5467fac17da0d6931a0cd, State: Running, Role: LEADER
I20260809 04:39:45.796381 20534 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3f65825cc5467fac17da0d6931a0cd" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33943 } }
I20260809 04:39:45.797083 20519 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:45.797681 20534 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa3f65825cc5467fac17da0d6931a0cd. Latest consensus state: current_term: 1 leader_uuid: "fa3f65825cc5467fac17da0d6931a0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3f65825cc5467fac17da0d6931a0cd" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33943 } } }
I20260809 04:39:45.797851 20534 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:45.798185 20536 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa3f65825cc5467fac17da0d6931a0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3f65825cc5467fac17da0d6931a0cd" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 33943 } } }
I20260809 04:39:45.798290 20536 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:45.799670 20557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:45.803107 20557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:45.804708 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:45.796551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47677 (local address 127.19.92.62:33943)
0809 04:39:45.796654 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:45.796658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:45.796672 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:39:45.796708 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:45.796711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:45.796753 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:45.796830 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:45.798447 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:45.799088 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:45.800113 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:45.800360 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:45.800471 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:45.801818 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:45.801821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:45.801824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:45.801849 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:45.803145 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:45.803564 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:45.803567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:45.803569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:45.803648 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:45.803895 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:45.803898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:45.803900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:45.804086 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:45.804198 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:45.804451 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 04:39:45.804560 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:39:45.809223 20557 catalog_manager.cc:1347] Generated new cluster ID: d021d2128f0542749f6c91575d9ae639
I20260809 04:39:45.809296 20557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:45.841379 20557 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:45.842427 20557 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:45.857899 20557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa3f65825cc5467fac17da0d6931a0cd: Generated new TSK 0
I20260809 04:39:45.858812 20557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:45.871903 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20418
2026-08-09T04:39:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1546 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:45 dist-test-slave-rmhg krb5kdc[20564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:45 dist-test-slave-rmhg krb5kdc[20564](info): set up 2 sockets
Aug 09 04:39:45 dist-test-slave-rmhg krb5kdc[20564](info): commencing operation
krb5kdc: starting...
W20260809 04:39:46.999820 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:47 dist-test-slave-rmhg krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:47.128475 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41625
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:47.243901 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:47.244189 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:47.244249 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:47.248288 20585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:47.248349 20585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:47.248377 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:47.248399 20585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:47.248420 20585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:47.253785 20585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:41625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:47.254892 20585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:47.255966 20585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:47.264747 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:47.264813 20591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:47.264855 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:47.265062 20585 server_base.cc:1034] running on GCE node
I20260809 04:39:47.265674 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:47.266762 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:47.267930 20585 hybrid_clock.cc:648] HybridClock initialized: now 1786250387267914 us; error 30 us; skew 500 ppm
Aug 09 04:39:47 dist-test-slave-rmhg krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250387, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:47.273758 20585 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:47.274650 20585 webserver.cc:460] Webserver started at http://127.19.92.62:34241/ using document root <none> and password file <none>
I20260809 04:39:47.275183 20585 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:47.275244 20585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:47.275465 20585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:47.277542 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "76a5825667444567ac183e2b774b00c5"
format_stamp: "Formatted at 2026-08-09 04:39:47 on dist-test-slave-rmhg"
server_key: "19e09ada1d989a859b9d0d8c13a90c8a"
server_key_iv: "5e06dcbb3d2c4ee1c698b7478ba44ead"
server_key_version: "encryptionkey@0"
I20260809 04:39:47.278059 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "76a5825667444567ac183e2b774b00c5"
format_stamp: "Formatted at 2026-08-09 04:39:47 on dist-test-slave-rmhg"
server_key: "19e09ada1d989a859b9d0d8c13a90c8a"
server_key_iv: "5e06dcbb3d2c4ee1c698b7478ba44ead"
server_key_version: "encryptionkey@0"
I20260809 04:39:47.281646 20585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:39:47.284009 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:47.285310 20585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:47.285430 20585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "76a5825667444567ac183e2b774b00c5"
format_stamp: "Formatted at 2026-08-09 04:39:47 on dist-test-slave-rmhg"
server_key: "19e09ada1d989a859b9d0d8c13a90c8a"
server_key_iv: "5e06dcbb3d2c4ee1c698b7478ba44ead"
server_key_version: "encryptionkey@0"
I20260809 04:39:47.285532 20585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:47.298728 20585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:47.301793 20585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:47.301962 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:47.308688 20585 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:41625
I20260809 04:39:47.308708 20681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:41625 every 8 connection(s)
I20260809 04:39:47.309626 20585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:47.312345 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:47.316130 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20585
I20260809 04:39:47.316280 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:47.316610 19824 external_mini_cluster.cc:1428] Setting key 33cab0f037b2b0afb1b727a6398326a0
I20260809 04:39:47.318282 20683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5: Bootstrap starting.
I20260809 04:39:47.320577 20683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:47.321348 20683 log.cc:824] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:47.323395 20683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5: No bootstrap required, opened a new log
I20260809 04:39:47.325582 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:47.318030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57123 (local address 127.19.92.62:41625)
0809 04:39:47.318476 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:47.318513 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:47.318548 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:39:47.319152 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:47.319169 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:47.319501 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:47.319767 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:47.320163 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:47.321218 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:47.322284 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:47.322769 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:47.322893 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:47.323909 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:47.323914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:47.323928 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:47.324304 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:47.324362 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:47.324384 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:47.324469 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:47.324625 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 04:39:47.324811 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260809 04:39:47.327229 20683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a5825667444567ac183e2b774b00c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41625 } }
I20260809 04:39:47.327474 20683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:47.327582 20683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a5825667444567ac183e2b774b00c5, State: Initialized, Role: FOLLOWER
I20260809 04:39:47.328049 20683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a5825667444567ac183e2b774b00c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41625 } }
I20260809 04:39:47.328212 20683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:47.328284 20683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:47.328337 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:47.328382 20683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:47.329273 20683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a5825667444567ac183e2b774b00c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41625 } }
I20260809 04:39:47.329627 20683 leader_election.cc:302] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a5825667444567ac183e2b774b00c5; no voters:
I20260809 04:39:47.329932 20683 leader_election.cc:288] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:47.330014 20693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:47.330261 20693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [term 1 LEADER]: Becoming Leader. State: Replica: 76a5825667444567ac183e2b774b00c5, State: Running, Role: LEADER
I20260809 04:39:47.330607 20693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a5825667444567ac183e2b774b00c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41625 } }
I20260809 04:39:47.331185 20683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:47.331634 20693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a5825667444567ac183e2b774b00c5. Latest consensus state: current_term: 1 leader_uuid: "76a5825667444567ac183e2b774b00c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a5825667444567ac183e2b774b00c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41625 } } }
I20260809 04:39:47.331785 20693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:47.332031 20693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76a5825667444567ac183e2b774b00c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a5825667444567ac183e2b774b00c5" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 41625 } } }
I20260809 04:39:47.332170 20693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:47.332206 20701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:47.335503 20701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:47.337047 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:47.330275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34033 (local address 127.19.92.62:41625)
0809 04:39:47.330373 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:47.330377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:47.332519 (+ 2142us) server_negotiation.cc:407] Connection header received
0809 04:39:47.332691 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:47.332694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:47.332738 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:47.332786 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:47.333768 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:47.334286 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:47.335406 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:47.335675 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:47.335766 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:47.336610 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:47.336615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:47.336618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:47.336632 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:47.336661 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:47.336673 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:47.336715 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:47.336859 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 04:39:47.336908 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 04:39:47.341264 20701 catalog_manager.cc:1347] Generated new cluster ID: a3eb25bf46f04670a14d7e1acad9c61d
I20260809 04:39:47.341322 20701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:47.352723 20701 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:47.353663 20701 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:47.358755 20701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76a5825667444567ac183e2b774b00c5: Generated new TSK 0
I20260809 04:39:47.359375 20701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:47.370736 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20585
2026-08-09T04:39:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1497 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:47 dist-test-slave-rmhg krb5kdc[20719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:47 dist-test-slave-rmhg krb5kdc[20719](info): set up 2 sockets
Aug 09 04:39:47 dist-test-slave-rmhg krb5kdc[20719](info): commencing operation
krb5kdc: starting...
W20260809 04:39:48.511966 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:48 dist-test-slave-rmhg krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:48.636181 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:45761
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:45761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:48.748503 20739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:48.748795 20739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:48.748855 20739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:48.752869 20739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:48.752933 20739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:48.752959 20739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:48.752981 20739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:48.753001 20739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:48.758247 20739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:45761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:45761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:48.759277 20739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:48.760254 20739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:48.768831 20744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:48.768831 20745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:48.768865 20739 server_base.cc:1034] running on GCE node
W20260809 04:39:48.768873 20747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:48.769810 20739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:48.770992 20739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:48.772197 20739 hybrid_clock.cc:648] HybridClock initialized: now 1786250388772179 us; error 29 us; skew 500 ppm
Aug 09 04:39:48 dist-test-slave-rmhg krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250388, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:48.775100 20739 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:48.776073 20739 webserver.cc:460] Webserver started at http://127.19.92.62:41933/ using document root <none> and password file <none>
I20260809 04:39:48.776659 20739 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:48.776731 20739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:48.776944 20739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:48.779135 20739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "3e2db199a5fa488da47dba7e0479bb77"
format_stamp: "Formatted at 2026-08-09 04:39:48 on dist-test-slave-rmhg"
server_key: "8a691d35f6027a2622ff5511d6c59aed"
server_key_iv: "8a4d1bcca28fc6e98334c73cbc18a6f5"
server_key_version: "encryptionkey@0"
I20260809 04:39:48.779703 20739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "3e2db199a5fa488da47dba7e0479bb77"
format_stamp: "Formatted at 2026-08-09 04:39:48 on dist-test-slave-rmhg"
server_key: "8a691d35f6027a2622ff5511d6c59aed"
server_key_iv: "8a4d1bcca28fc6e98334c73cbc18a6f5"
server_key_version: "encryptionkey@0"
I20260809 04:39:48.783761 20739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:39:48.786428 20756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:48.787711 20739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:39:48.787845 20739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "3e2db199a5fa488da47dba7e0479bb77"
format_stamp: "Formatted at 2026-08-09 04:39:48 on dist-test-slave-rmhg"
server_key: "8a691d35f6027a2622ff5511d6c59aed"
server_key_iv: "8a4d1bcca28fc6e98334c73cbc18a6f5"
server_key_version: "encryptionkey@0"
I20260809 04:39:48.787962 20739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:48.792732 20739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:48.795553 20739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:48.795730 20739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:48.802748 20739 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:45761
I20260809 04:39:48.802789 20831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:45761 every 8 connection(s)
I20260809 04:39:48.803680 20739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:48.806550 20833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:48.812104 20833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77: Bootstrap starting.
I20260809 04:39:48.813047 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20739
I20260809 04:39:48.813194 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:48.813453 19824 external_mini_cluster.cc:1428] Setting key a043371fdc28500c08d57f3bfcefb0c7
I20260809 04:39:48.814886 20833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:48.815805 20833 log.cc:824] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:48.818080 20833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77: No bootstrap required, opened a new log
I20260809 04:39:48.818795 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:48.814552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44165 (local address 127.19.92.62:45761)
0809 04:39:48.815072 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:48.815093 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:48.815141 (+ 48us) server_negotiation.cc:407] Connection header received
0809 04:39:48.816077 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:48.816103 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:48.816422 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:48.816777 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:48.817446 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:48.817451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:48.817466 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:48.817477 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:48.817538 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:48.817562 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:48.817669 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:48.817737 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 04:39:48.817891 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":156,"threads_started":1}
I20260809 04:39:48.821763 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:48.822227 20833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2db199a5fa488da47dba7e0479bb77" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45761 } }
I20260809 04:39:48.822516 20833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:39:48.822589 20833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2db199a5fa488da47dba7e0479bb77, State: Initialized, Role: FOLLOWER
I20260809 04:39:48.823068 20833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2db199a5fa488da47dba7e0479bb77" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45761 } }
I20260809 04:39:48.823247 20833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:39:48.823325 20833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:39:48.823428 20833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:48.824582 20833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2db199a5fa488da47dba7e0479bb77" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45761 } }
I20260809 04:39:48.824965 20833 leader_election.cc:302] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e2db199a5fa488da47dba7e0479bb77; no voters:
I20260809 04:39:48.825337 20833 leader_election.cc:288] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:39:48.825438 20846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:48.825649 20846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2db199a5fa488da47dba7e0479bb77, State: Running, Role: LEADER
I20260809 04:39:48.825865 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:48.824026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55069 (local address 127.19.92.62:45761)
0809 04:39:48.824148 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:48.824152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:48.824308 (+ 156us) server_negotiation.cc:407] Connection header received
0809 04:39:48.824462 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:48.824464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:48.824512 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:48.824573 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:48.825242 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:48.825246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:48.825248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:39:48.825260 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:48.825289 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:39:48.825299 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:48.825377 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:48.825712 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 04:39:48.825753 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:39:48.826023 20846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2db199a5fa488da47dba7e0479bb77" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45761 } }
I20260809 04:39:48.826732 20833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:48.827944 20847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2db199a5fa488da47dba7e0479bb77. Latest consensus state: current_term: 1 leader_uuid: "3e2db199a5fa488da47dba7e0479bb77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2db199a5fa488da47dba7e0479bb77" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45761 } } }
I20260809 04:39:48.827999 20848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e2db199a5fa488da47dba7e0479bb77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2db199a5fa488da47dba7e0479bb77" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 45761 } } }
I20260809 04:39:48.828138 20847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:48.828214 20848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:48.829111 20859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:48.831830 20859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:48.837386 20859 catalog_manager.cc:1347] Generated new cluster ID: 320631f35c424db9a3574df63f6227c5
I20260809 04:39:48.837456 20859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:48.850751 20859 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:48.851089 20859 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:48.860569 20859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e2db199a5fa488da47dba7e0479bb77: Generated new TSK 0
I20260809 04:39:48.861426 20859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:39:48.865824 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20739
2026-08-09T04:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1497 ms)
[----------] 18 tests from AuthTokenIssuingTest (27199 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:48 dist-test-slave-rmhg krb5kdc[20873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:48 dist-test-slave-rmhg krb5kdc[20873](info): set up 2 sockets
Aug 09 04:39:48 dist-test-slave-rmhg krb5kdc[20873](info): commencing operation
krb5kdc: starting...
W20260809 04:39:50.007074 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:50.131735 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36503
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36503,127.19.92.61:45195,127.19.92.60:33343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:50.245052 20893 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:39:50.245421 20893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:50.245473 20893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:50.245532 20893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:50.249549 20893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:50.249612 20893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:50.249639 20893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:50.249660 20893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:50.249682 20893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:50.255064 20893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36503,127.19.92.61:45195,127.19.92.60:33343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:50.256217 20893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:50.257359 20893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:50.265951 20899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:50.266141 20902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:50.265946 20900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:50.266405 20893 server_base.cc:1034] running on GCE node
I20260809 04:39:50.266978 20893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:50.268144 20893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:50.269305 20893 hybrid_clock.cc:648] HybridClock initialized: now 1786250390269274 us; error 50 us; skew 500 ppm
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:50.272161 20893 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:50.273186 20893 webserver.cc:460] Webserver started at http://127.19.92.62:35723/ using document root <none> and password file <none>
I20260809 04:39:50.273769 20893 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:50.273840 20893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:50.274044 20893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:50.276151 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "004f713119ea4037a94236f1a1ca1926"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "b51759f1a34a13714d3305166f4942c0"
server_key_iv: "bfb49237ac221edf72fb8bc8152f8a8f"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.276710 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "004f713119ea4037a94236f1a1ca1926"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "b51759f1a34a13714d3305166f4942c0"
server_key_iv: "bfb49237ac221edf72fb8bc8152f8a8f"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.280521 20893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:39:50.282991 20914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:50.284220 20893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:39:50.284337 20893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "004f713119ea4037a94236f1a1ca1926"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "b51759f1a34a13714d3305166f4942c0"
server_key_iv: "bfb49237ac221edf72fb8bc8152f8a8f"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.284451 20893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:50.302299 20893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:50.305348 20893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:50.305547 20893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:50.312768 20893 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:36503
I20260809 04:39:50.313054 20993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:36503 every 8 connection(s)
I20260809 04:39:50.313822 20893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:50.316612 20995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:50.318527 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 20893
I20260809 04:39:50.318667 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:50.318941 19824 external_mini_cluster.cc:1428] Setting key 9f3d73db8960395b67192f3c456368ea
I20260809 04:39:50.320573 20995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.92.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.61@KRBTEST.COM" created.
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:50.336127 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.323072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36503 (local address 127.0.0.1:39056)
0809 04:39:50.323612 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.323658 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.324597 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.325296 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.325301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.325727 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.325979 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.325993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.327122 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.327125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.327909 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.327919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.328600 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.328604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.328748 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.329307 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.329325 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.331046 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.332971 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.332982 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.332994 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.333335 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.333683 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.333687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.333689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.333808 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.334417 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.334423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.334571 (+ 148us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.334725 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.335132 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":125,"thread_start_us":150,"threads_started":1}
I20260809 04:39:50.336138 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.324102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.19.92.62:36503)
0809 04:39:50.324421 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.324432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.324489 (+ 57us) server_negotiation.cc:407] Connection header received
0809 04:39:50.324892 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.324896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.325081 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.325296 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.326123 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.327002 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.328036 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.328452 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.328599 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.331208 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.331215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.331231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.331259 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.332848 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.333453 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.333459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.333460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.333578 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.333896 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.333899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.333901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.334305 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.334429 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.334818 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.335068 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":199,"spinlock_wait_cycles":4352,"thread_start_us":93,"threads_started":1}
I20260809 04:39:50.339248 20995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:50.340004 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.339730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33424)
0809 04:39:50.339857 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.339929 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 04:39:50.340224 20915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
I20260809 04:39:50.341113 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.340901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33432)
0809 04:39:50.341009 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.341055 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:39:50.341295 20995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:45195: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab.
Entry for principal kudu/127.19.92.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab.
I20260809 04:39:50.346298 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.61:45195
--webserver_interface=127.19.92.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab
--principal=kudu/127.19.92.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36503,127.19.92.61:45195,127.19.92.60:33343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:39:50.390573 20995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } attempt: 1
I20260809 04:39:50.391497 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.391149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33446)
0809 04:39:50.391319 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.391392 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 04:39:50.392270 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.392024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33448)
0809 04:39:50.392134 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.392192 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 04:39:50.392411 20995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:45195: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
I20260809 04:39:50.460613 20995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } attempt: 2
I20260809 04:39:50.461544 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.461211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33454)
0809 04:39:50.461372 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.461442 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 04:39:50.462260 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.462031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33464)
0809 04:39:50.462151 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.462198 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 04:39:50.462504 20995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:45195: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45195: connect: Connection refused (error 111)
W20260809 04:39:50.463551 21003 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:39:50.463810 21003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:50.463832 21003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:50.463874 21003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:50.467897 21003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:50.467954 21003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:50.467973 21003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:50.467993 21003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:50.468008 21003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:50.473387 21003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36503,127.19.92.61:45195,127.19.92.60:33343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab
--principal=kudu/127.19.92.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.61:45195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.92.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:50.474476 21003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:50.475498 21003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:50.484232 21015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:50.484447 21003 server_base.cc:1034] running on GCE node
W20260809 04:39:50.484232 21011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:50.484241 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:50.485208 21003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:50.486359 21003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:50.487525 21003 hybrid_clock.cc:648] HybridClock initialized: now 1786250390487497 us; error 43 us; skew 500 ppm
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:50.490367 21003 init.cc:377] Logged in from keytab as kudu/127.19.92.61@KRBTEST.COM (short username kudu)
I20260809 04:39:50.491326 21003 webserver.cc:460] Webserver started at http://127.19.92.61:35057/ using document root <none> and password file <none>
I20260809 04:39:50.491931 21003 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:50.492031 21003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:50.492261 21003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:50.494432 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/data/instance:
uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "d28567dc92066cf780b9fddfbaba49c3"
server_key_iv: "fa7668e7b97559ebbfd107d7a12d3acf"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.495021 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/wal/instance:
uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "d28567dc92066cf780b9fddfbaba49c3"
server_key_iv: "fa7668e7b97559ebbfd107d7a12d3acf"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.498878 21003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 04:39:50.501497 21024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:50.502766 21003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:39:50.502893 21003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/wal
uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "d28567dc92066cf780b9fddfbaba49c3"
server_key_iv: "fa7668e7b97559ebbfd107d7a12d3acf"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.503008 21003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:50.532245 21003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:50.535316 21003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:50.535511 21003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:50.542177 21003 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.61:45195
I20260809 04:39:50.542438 21096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.61:45195 every 8 connection(s)
I20260809 04:39:50.543136 21003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
I20260809 04:39:50.544571 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21003
I20260809 04:39:50.544719 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-1/wal/instance
I20260809 04:39:50.545030 19824 external_mini_cluster.cc:1428] Setting key f8af4df6b82c46ddaa93d7f5909063e9
I20260809 04:39:50.546443 21098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:50.550354 21098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.92.60@KRBTEST.COM; defaulting to no policy
I20260809 04:39:50.553659 20995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.60@KRBTEST.COM" created.
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:50.564383 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.554046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33474)
0809 04:39:50.554166 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.554179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.554249 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.555096 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.555098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.555283 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.555405 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.555412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.557022 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.557026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.557583 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.557592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.558268 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.558272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.558363 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.558927 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.558943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.560206 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.562303 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.562306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.562308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.562568 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.562960 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.562964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.562966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.563065 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.563814 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.563817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.563924 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.564032 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.564195 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":7,"spinlock_wait_cycles":1920}
I20260809 04:39:50.564988 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.552245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.19.92.62:36503)
0809 04:39:50.552371 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.552374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.553421 (+ 1047us) server_negotiation.cc:407] Connection header received
0809 04:39:50.554658 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.554661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.554711 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.554791 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.555844 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.556609 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.557971 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.558297 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.558393 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.561007 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.561011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.561014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.561042 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.563093 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.563591 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.563595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.563596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.563678 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.563959 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.563963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.563965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.564211 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.564299 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.564726 (+ 427us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.564835 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":1792}
I20260809 04:39:50.565773 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.554196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.19.92.61:45195)
0809 04:39:50.554547 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.554555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.554589 (+ 34us) server_negotiation.cc:407] Connection header received
0809 04:39:50.554732 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.554737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.554944 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.555099 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.555607 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.556895 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.557697 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.558139 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.558638 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.560339 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.560345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.560355 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.560377 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.562178 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.562688 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.562695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.562702 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.562846 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.563167 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.563170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.563172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.563693 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.563851 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.564180 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.564735 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":255,"spinlock_wait_cycles":3968,"thread_start_us":136,"threads_started":1}
I20260809 04:39:50.565773 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.552753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36503 (local address 127.0.0.1:39064)
0809 04:39:50.553280 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.553323 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.554277 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.554862 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.554869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.555174 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.555708 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.555721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.556732 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.556737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.557847 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.557859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.558452 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.558457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.558626 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.559141 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.559162 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.560859 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.563201 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.563211 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.563217 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.563486 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.563779 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.563782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.563784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.563882 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.564349 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.564355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.564482 (+ 127us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.565123 (+ 641us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.565323 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":22,"spinlock_wait_cycles":1792,"thread_start_us":145,"threads_started":1}
I20260809 04:39:50.567227 21098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:50.567816 20995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:50.568439 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.568220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44724)
0809 04:39:50.568322 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.568372 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 04:39:50.569192 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.568906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44736)
0809 04:39:50.569072 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.569115 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260809 04:39:50.569365 20995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:33343: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab.
Entry for principal kudu/127.19.92.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab.
I20260809 04:39:50.574263 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.60:33343
--webserver_interface=127.19.92.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab
--principal=kudu/127.19.92.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36503,127.19.92.61:45195,127.19.92.60:33343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:39:50.575515 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.567698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33484)
0809 04:39:50.568124 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.568139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.568225 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.568466 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.568469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.568635 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.568739 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.568746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.569670 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.569672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.570307 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.570316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.570803 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.570805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.570912 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.571401 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.571413 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.572601 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.574197 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.574200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.574202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.574408 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.574675 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.574677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.574678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.574754 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.575128 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.575130 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.575212 (+ 82us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.575291 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.575389 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":1}
I20260809 04:39:50.575645 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.567756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.19.92.61:45195)
0809 04:39:50.567862 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.567865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.568298 (+ 433us) server_negotiation.cc:407] Connection header received
0809 04:39:50.568322 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.568325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.568360 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.568429 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.568863 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.569585 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.570465 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.570714 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.570796 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.572716 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.572719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.572721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.572742 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.574090 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.574498 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.574501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.574502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.574586 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.574832 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.574835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.574837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.575044 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.575133 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.575366 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.575478 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":42}
I20260809 04:39:50.576244 21098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:50.576927 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.576645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44746)
0809 04:39:50.576754 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.576844 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 04:39:50.577571 21025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
I20260809 04:39:50.578424 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.578198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44748)
0809 04:39:50.578318 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.578369 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 04:39:50.578567 21098 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:33343: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
I20260809 04:39:50.600554 20995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } attempt: 1
I20260809 04:39:50.601480 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.601111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44760)
0809 04:39:50.601309 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.601382 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260809 04:39:50.602239 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.601967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44776)
0809 04:39:50.602110 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.602169 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 04:39:50.602376 20995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:33343: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
I20260809 04:39:50.627880 21098 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } attempt: 1
I20260809 04:39:50.628814 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.628468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44782)
0809 04:39:50.628644 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.628718 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 04:39:50.629520 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.629298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44788)
0809 04:39:50.629401 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.629455 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 04:39:50.629655 21098 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:33343: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
I20260809 04:39:50.677562 20995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } attempt: 2
I20260809 04:39:50.678505 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.678130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44802)
0809 04:39:50.678320 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.678403 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 04:39:50.679306 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.679042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44808)
0809 04:39:50.679176 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.679235 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 04:39:50.679446 20995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:33343: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
W20260809 04:39:50.690475 21104 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:39:50.690784 21104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:50.690825 21104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:50.690886 21104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:50.695014 21104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:50.695080 21104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:50.695099 21104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:50.695125 21104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:50.695153 21104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:50.697800 21098 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } attempt: 2
I20260809 04:39:50.698652 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.698334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44812)
0809 04:39:50.698490 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.698558 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 04:39:50.699288 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.699096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44818)
0809 04:39:50.699186 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.699232 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 04:39:50.699424 21098 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:33343: Network error: Client connection negotiation failed: client connection to 127.19.92.60:33343: connect: Connection refused (error 111)
I20260809 04:39:50.700954 21104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36503,127.19.92.61:45195,127.19.92.60:33343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab
--principal=kudu/127.19.92.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.60:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.92.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:50.702135 21104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:50.703155 21104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:50.711467 21114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:50.711505 21113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:50.711476 21117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:50.711715 21104 server_base.cc:1034] running on GCE node
I20260809 04:39:50.712324 21104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:50.713409 21104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:50.714568 21104 hybrid_clock.cc:648] HybridClock initialized: now 1786250390714535 us; error 46 us; skew 500 ppm
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:50.717067 21104 init.cc:377] Logged in from keytab as kudu/127.19.92.60@KRBTEST.COM (short username kudu)
I20260809 04:39:50.717932 21104 webserver.cc:460] Webserver started at http://127.19.92.60:41195/ using document root <none> and password file <none>
I20260809 04:39:50.718434 21104 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:50.718498 21104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:50.718659 21104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:50.720628 21104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/data/instance:
uuid: "04946764b92646ab9f6e1172736219e9"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "5cba67f7e9c25a0236d3e1af654516ac"
server_key_iv: "a92681865eb24bdcc9bc8619afe1ca8d"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.721235 21104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/wal/instance:
uuid: "04946764b92646ab9f6e1172736219e9"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "5cba67f7e9c25a0236d3e1af654516ac"
server_key_iv: "a92681865eb24bdcc9bc8619afe1ca8d"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.724850 21104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:39:50.727353 21126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:50.728581 21104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:39:50.728683 21104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/wal
uuid: "04946764b92646ab9f6e1172736219e9"
format_stamp: "Formatted at 2026-08-09 04:39:50 on dist-test-slave-rmhg"
server_key: "5cba67f7e9c25a0236d3e1af654516ac"
server_key_iv: "a92681865eb24bdcc9bc8619afe1ca8d"
server_key_version: "encryptionkey@0"
I20260809 04:39:50.728794 21104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:50.754338 21104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:50.757289 21104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:50.757468 21104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:50.764333 21104 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.60:33343
I20260809 04:39:50.764614 21197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.60:33343 every 8 connection(s)
I20260809 04:39:50.765347 21104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
I20260809 04:39:50.768141 21199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:50.772106 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21104
I20260809 04:39:50.771899 21199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:50.772245 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250097044341-19824-0/minicluster-data/master-2/wal/instance
I20260809 04:39:50.772537 19824 external_mini_cluster.cc:1428] Setting key 76904dddc3e870281cf9cb854f6f3c86
I20260809 04:39:50.778602 20995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } attempt: 3
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
I20260809 04:39:50.785522 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.773969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.19.92.62:36503)
0809 04:39:50.774096 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.774099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.774568 (+ 469us) server_negotiation.cc:407] Connection header received
0809 04:39:50.775398 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.775401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.775458 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.775547 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.776404 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.776918 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.777960 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.778194 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.778285 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.781338 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.781343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.781346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.781375 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.783023 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.783787 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.783791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.783793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.783872 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.784187 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.784191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.784193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.784392 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.784497 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.784941 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.785062 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:39:50.786366 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.773891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36503 (local address 127.0.0.1:39072)
0809 04:39:50.774388 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.774436 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.775158 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.775639 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.775646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.775992 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.776254 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.776267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.777038 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.777041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.777836 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.777845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.778345 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.778349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.778577 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.779385 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.779400 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.781115 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.783326 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.783336 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.783347 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.783652 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.783980 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.783982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.783984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.784093 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.784549 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.784555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.784717 (+ 162us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.784873 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.785237 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":1,"spinlock_wait_cycles":6400,"thread_start_us":127,"threads_started":1}
I20260809 04:39:50.786607 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.773891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.19.92.60:33343)
0809 04:39:50.774333 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.774351 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.774391 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:39:50.775146 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.775171 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.775412 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.775564 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.776393 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.777282 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.778095 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.778517 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.778608 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.781553 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.781560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.781574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.781607 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.783343 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.783764 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.783771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.783772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.783906 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.784233 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.784236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.784238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.784750 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.784958 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.785359 (+ 401us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.785497 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":322,"spinlock_wait_cycles":12928,"thread_start_us":162,"threads_started":1}
I20260809 04:39:50.787971 21199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:50.789722 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.779055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44838)
0809 04:39:50.779169 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.779181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.779245 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.779686 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.779690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.779862 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.779978 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.779985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.781604 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.781608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.782330 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.782342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.782839 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.782843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.782940 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.783323 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.783338 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.784712 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.788151 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.788155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.788157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.788383 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.788683 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.788686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.788688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.788795 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.789231 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.789234 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.789331 (+ 97us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.789431 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.789568 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 04:39:50.789763 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.779121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.19.92.60:33343)
0809 04:39:50.779451 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.779460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.779472 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:39:50.779502 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.779504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.779544 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.779631 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.780734 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.781448 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.782478 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.782720 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.782816 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.784847 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.784851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.784854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.784874 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.788035 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.788505 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.788508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.788510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.788604 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.788876 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.788879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.788881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.789136 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.789240 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.789494 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.789604 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
I20260809 04:39:50.790558 21098 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } attempt: 3
I20260809 04:39:50.793116 20995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926: Bootstrap starting.
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
I20260809 04:39:50.795997 20995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926: Neither blocks nor log segments found. Creating new log.
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:50.797094 20995 log.cc:824] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:50.797307 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.788576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.19.92.61:45195)
0809 04:39:50.788696 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.788700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.788713 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:39:50.788827 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.788830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.788883 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.788956 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.789388 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.790096 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.791125 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.791501 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.791573 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.793872 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.793876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.793879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.793905 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.795661 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.796063 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.796066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.796068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.796149 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.796400 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.796403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.796405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.796587 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.796676 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.797012 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.797127 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:39:50.797816 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.788475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45195 (local address 127.0.0.1:33488)
0809 04:39:50.788653 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.788665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.788723 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.788975 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.788978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.789170 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.789265 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.789271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.790207 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.790211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.791004 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.791014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.791714 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.791717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.791804 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.792267 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.792281 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.793717 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.795766 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.795769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.795771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.795968 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.796235 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.796238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.796240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.796318 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.796720 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.796723 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.796821 (+ 98us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.797522 (+ 701us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.797661 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260809 04:39:50.798574 21199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:50.799381 20995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926: No bootstrap required, opened a new log
I20260809 04:39:50.800515 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.789669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.19.92.62:36503)
0809 04:39:50.789767 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.789770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.790062 (+ 292us) server_negotiation.cc:407] Connection header received
0809 04:39:50.790097 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.790100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.790137 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.790195 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.791012 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.791692 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.792749 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.793075 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.793209 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.797107 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.797111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.797114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.797138 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.798904 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.799351 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.799355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.799357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.799450 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.799722 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.799725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.799727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.799897 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.800074 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.800264 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.800366 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:39:50.801502 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.790901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44844)
0809 04:39:50.791135 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.791148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.791216 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.791659 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.791662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.791833 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.791927 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.791933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.792925 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.792929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.793659 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.793669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.794308 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.794312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.794403 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.794799 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.794813 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.796941 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.799947 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.799950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.799952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.800190 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.800513 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.800516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.800517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.800601 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.800991 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.800994 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.801091 (+ 97us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.801233 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.801366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260809 04:39:50.801832 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.791372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.19.92.60:33343)
0809 04:39:50.791458 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.791461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.791471 (+ 10us) server_negotiation.cc:407] Connection header received
0809 04:39:50.791500 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.791503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.791539 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.792047 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.792128 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.792800 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.793874 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.794205 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.794301 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.797088 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.797092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.797094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.797118 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.799824 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.800310 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.800313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.800314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.800394 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.800700 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.800703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.800704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.800890 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.800981 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.801385 (+ 404us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.801504 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 04:39:50.803915 20995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } }
I20260809 04:39:50.804152 20995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004f713119ea4037a94236f1a1ca1926, State: Initialized, Role: FOLLOWER
I20260809 04:39:50.804502 20995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } }
I20260809 04:39:50.804801 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99: Bootstrap starting.
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
I20260809 04:39:50.805917 20995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 04:39:50.807129 20922 tablet.cc:2369] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:50.808300 21098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99: Neither blocks nor log segments found. Creating new log.
Aug 09 04:39:50 dist-test-slave-rmhg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
I20260809 04:39:50.809580 21098 log.cc:824] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:50.810915 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.801268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44846 (local address 127.19.92.60:33343)
0809 04:39:50.801358 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.801362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.801373 (+ 11us) server_negotiation.cc:407] Connection header received
0809 04:39:50.801399 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.801402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.801437 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.801513 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.801933 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.802647 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.803609 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.803915 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.803980 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.806134 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.806138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.806140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.806161 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.807765 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.808216 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.808219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.808221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.808290 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.808560 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.808563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.808565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.808747 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.808815 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.810643 (+ 1828us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.810760 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260809 04:39:50.811789 21226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:50.811863 21226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:50.811856 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99: No bootstrap required, opened a new log
I20260809 04:39:50.814549 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.802323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.19.92.61:45195)
0809 04:39:50.802725 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.802729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.802876 (+ 147us) server_negotiation.cc:407] Connection header received
0809 04:39:50.803048 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.803051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.803092 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.803135 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.803886 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.805001 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.806280 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.806873 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.806964 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.810111 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.810116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.810119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.810146 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.812460 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.813167 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.813172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.813174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.813266 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.813560 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.813563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.813565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.813766 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.813909 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.814286 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.814404 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260809 04:39:50.811230 20893 master_runner.cc:186] Error getting master registration for 127.19.92.61:45195: OK, instance_id { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" instance_seqno: 1786250390540341 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:39:50.815182 20893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:39:50.815562 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.799034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:33343 (local address 127.0.0.1:44846)
0809 04:39:50.799160 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:50.799173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:50.799242 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:50.801553 (+ 2311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:50.801556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:50.801732 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:50.801819 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.801826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.802762 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.802764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.803491 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:50.803502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.804098 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.804101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:50.804173 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.804571 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:50.804585 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:50.806015 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:50.807897 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.807900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.807902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.808119 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.808378 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:50.808381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:50.808383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:50.808471 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:50.810369 (+ 1898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:50.810372 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:50.810482 (+ 110us) client_negotiation.cc:769] Sending connection context
0809 04:39:50.815256 (+ 4774us) client_negotiation.cc:240] Negotiation successful
0809 04:39:50.815405 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 04:39:50.815812 21098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } }
I20260809 04:39:50.816045 21098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a0d28d0ec448f7bd4ffc1fd4f3bd99, State: Initialized, Role: FOLLOWER
I20260809 04:39:50.816391 21098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } }
I20260809 04:39:50.816907 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.817108 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:50.817803 21098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:50.818306 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9: Bootstrap starting.
I20260809 04:39:50.821365 21207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.821521 21207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:50.822690 21199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9: Neither blocks nor log segments found. Creating new log.
W20260809 04:39:50.823004 21247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:50.823115 21247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:50.823534 21199 log.cc:824] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:50.823570 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:50.825350 20954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "004f713119ea4037a94236f1a1ca1926"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084
I20260809 04:39:50.825543 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9: No bootstrap required, opened a new log
I20260809 04:39:50.825582 20954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 04:39:50.825690 20954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:50.827538 20954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } }
I20260809 04:39:50.828495 20954 leader_election.cc:288] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [CANDIDATE]: Term 1 election: Requested vote from peers 48a0d28d0ec448f7bd4ffc1fd4f3bd99 (127.19.92.61:45195), 04946764b92646ab9f6e1172736219e9 (127.19.92.60:33343)
I20260809 04:39:50.829515 21199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } }
I20260809 04:39:50.829787 21199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04946764b92646ab9f6e1172736219e9, State: Initialized, Role: FOLLOWER
I20260809 04:39:50.830135 21199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } }
I20260809 04:39:50.827122 21003 master_runner.cc:186] Error getting master registration for 127.19.92.60:33343: OK, instance_id { permanent_uuid: "04946764b92646ab9f6e1172736219e9" instance_seqno: 1786250390762405 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:39:50.830366 21003 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:39:50.831312 21062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "004f713119ea4037a94236f1a1ca1926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99"
I20260809 04:39:50.831526 21199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:50.831537 21062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:50.832826 21062 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 004f713119ea4037a94236f1a1ca1926 in term 1.
I20260809 04:39:50.834645 20915 leader_election.cc:302] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004f713119ea4037a94236f1a1ca1926, 48a0d28d0ec448f7bd4ffc1fd4f3bd99; no voters:
I20260809 04:39:50.835269 21207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:50.835489 21207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [term 1 LEADER]: Becoming Leader. State: Replica: 004f713119ea4037a94236f1a1ca1926, State: Running, Role: LEADER
I20260809 04:39:50.835817 21207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } }
W20260809 04:39:50.836918 21266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:50.836997 21266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:50.837307 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.837437 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:50.833645 21163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "004f713119ea4037a94236f1a1ca1926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04946764b92646ab9f6e1172736219e9"
I20260809 04:39:50.839779 21163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:50.841122 21163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 004f713119ea4037a94236f1a1ca1926 in term 1.
I20260809 04:39:50.844201 21207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004f713119ea4037a94236f1a1ca1926. Latest consensus state: current_term: 1 leader_uuid: "004f713119ea4037a94236f1a1ca1926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.844336 21207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:50.846801 21277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:50.847728 21277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:50.848572 21104 master_runner.cc:263] No leader master found from master 04946764b92646ab9f6e1172736219e9
I20260809 04:39:50.848635 21104 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:39:50.863993 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.844649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.19.92.61:45195)
0809 04:39:50.844776 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.844780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.852052 (+ 7272us) server_negotiation.cc:407] Connection header received
0809 04:39:50.852249 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.852253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.852302 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.852349 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.855796 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.856596 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.857907 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.858258 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.858331 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.860290 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.860294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.860297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.860327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.862086 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.862585 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.862589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.862591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.862676 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.862974 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.862978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.862980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.863198 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.863305 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.863674 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.863814 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 04:39:50.866796 21163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [term 1 FOLLOWER]: Refusing update from remote peer 004f713119ea4037a94236f1a1ca1926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:50.867516 21269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:50.871637 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004f713119ea4037a94236f1a1ca1926. Latest consensus state: current_term: 1 leader_uuid: "004f713119ea4037a94236f1a1ca1926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.871766 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:50.872722 21062 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [term 1 FOLLOWER]: Refusing update from remote peer 004f713119ea4037a94236f1a1ca1926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:50.873338 21269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:50.877379 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004f713119ea4037a94236f1a1ca1926. Latest consensus state: current_term: 1 leader_uuid: "004f713119ea4037a94236f1a1ca1926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.877508 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:50.893052 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.844940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.19.92.62:36503)
0809 04:39:50.845045 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.845049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.861442 (+ 16393us) server_negotiation.cc:407] Connection header received
0809 04:39:50.861623 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.861626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.861678 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.861726 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.862568 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.863370 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.879087 (+ 15717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.879465 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.879542 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.889227 (+ 9685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.889232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.889235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.889270 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.891041 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.891570 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.891574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.891575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.891663 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.891961 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.891964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.891966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.892218 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.892344 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.892746 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.892878 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:39:50.894253 21269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "004f713119ea4037a94236f1a1ca1926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.894402 21269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:50.896308 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "004f713119ea4037a94236f1a1ca1926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.896457 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:50.896853 21282 mvcc.cc:205] Tried to move back new op lower bound from 7316481600965308416 to 7316481600866975744. Current Snapshot: MvccSnapshot[applied={T|T < 7316481600965308416}]
I20260809 04:39:50.897965 21207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "004f713119ea4037a94236f1a1ca1926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.898097 21207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:50.899416 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "004f713119ea4037a94236f1a1ca1926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004f713119ea4037a94236f1a1ca1926" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36503 } } peers { permanent_uuid: "48a0d28d0ec448f7bd4ffc1fd4f3bd99" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45195 } } peers { permanent_uuid: "04946764b92646ab9f6e1172736219e9" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 33343 } } }
I20260809 04:39:50.899547 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:50.900906 21287 mvcc.cc:205] Tried to move back new op lower bound from 7316481600965308416 to 7316481600866975744. Current Snapshot: MvccSnapshot[applied={T|T < 7316481600965308416}]
I20260809 04:39:50.902422 21283 mvcc.cc:205] Tried to move back new op lower bound from 7316481600965308416 to 7316481600866975744. Current Snapshot: MvccSnapshot[applied={T|T < 7316481600965308416}]
I20260809 04:39:50.902971 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:50.844637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.19.92.60:33343)
0809 04:39:50.844815 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.844819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.865422 (+ 20603us) server_negotiation.cc:407] Connection header received
0809 04:39:50.877898 (+ 12476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.877902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.877967 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.878018 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.880125 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.880872 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.882085 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:50.882426 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:50.882499 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:50.890573 (+ 8074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:50.890577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:50.890580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:50.890609 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:50.892178 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.901137 (+ 8959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.901142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.901144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.901298 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:50.901678 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:50.901682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:50.901683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:50.901944 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:50.902061 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:50.902661 (+ 600us) server_negotiation.cc:299] Negotiation successful
0809 04:39:50.902804 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260809 04:39:50.908943 21277 catalog_manager.cc:1347] Generated new cluster ID: 738729a805684589a03beb5b3de730aa
I20260809 04:39:50.909024 21277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:50.929586 21277 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:50.930972 21277 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:50.947829 21277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 004f713119ea4037a94236f1a1ca1926: Generated new TSK 0
I20260809 04:39:50.948742 21277 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:39:50.977649 21000 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:50.974367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.19.92.62:36503)
0809 04:39:50.974498 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.974503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.976469 (+ 1966us) server_negotiation.cc:407] Connection header received
0809 04:39:50.976679 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.976684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.976739 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.976795 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.977510 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39106: BlockingRecv error: recv got EOF from 127.0.0.1:39106 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 04:39:50.979118 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:50.975216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.19.92.61:45195)
0809 04:39:50.975328 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.975332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.978178 (+ 2846us) server_negotiation.cc:407] Connection header received
0809 04:39:50.978354 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.978357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.978408 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.978458 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.978987 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33504: BlockingRecv error: recv got EOF from 127.0.0.1:33504 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 04:39:50.980500 21203 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:50.975860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.19.92.60:33343)
0809 04:39:50.975986 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.975990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.979597 (+ 3607us) server_negotiation.cc:407] Connection header received
0809 04:39:50.979767 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.979770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.979820 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.979869 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.980379 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44862: BlockingRecv error: recv got EOF from 127.0.0.1:44862 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 04:39:50.985373 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:50.983720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.19.92.61:45195)
0809 04:39:50.983826 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.983830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.984286 (+ 456us) server_negotiation.cc:407] Connection header received
0809 04:39:50.984542 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.984545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.984627 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.984683 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.985271 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33510: BlockingRecv error: recv got EOF from 127.0.0.1:33510 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 04:39:50.990835 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:50.989263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.19.92.61:45195)
0809 04:39:50.989370 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.989374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.989870 (+ 496us) server_negotiation.cc:407] Connection header received
0809 04:39:50.990091 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.990094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.990147 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.990200 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.990726 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33520: BlockingRecv error: recv got EOF from 127.0.0.1:33520 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 04:39:50.996671 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:50.995133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.19.92.61:45195)
0809 04:39:50.995239 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:50.995243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:50.995727 (+ 484us) server_negotiation.cc:407] Connection header received
0809 04:39:50.995934 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:50.995937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:50.996002 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:50.996057 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:50.996571 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33522: BlockingRecv error: recv got EOF from 127.0.0.1:33522 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 04:39:51.004377 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:51.002982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.19.92.61:45195)
0809 04:39:51.003107 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:51.003111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:51.003551 (+ 440us) server_negotiation.cc:407] Connection header received
0809 04:39:51.003713 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:51.003716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:51.003761 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:51.003806 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:51.004277 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33532: BlockingRecv error: recv got EOF from 127.0.0.1:33532 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 04:39:51.016762 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:51.014976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.19.92.61:45195)
0809 04:39:51.015096 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:51.015101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:51.015698 (+ 597us) server_negotiation.cc:407] Connection header received
0809 04:39:51.015901 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:51.015905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:51.015993 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:51.016047 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:51.016634 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33542: BlockingRecv error: recv got EOF from 127.0.0.1:33542 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 04:39:51.036399 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:51.035290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.19.92.61:45195)
0809 04:39:51.035430 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:51.035434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:51.035544 (+ 110us) server_negotiation.cc:407] Connection header received
0809 04:39:51.035671 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:51.035675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:51.035734 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:51.035844 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:51.036273 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33552: BlockingRecv error: recv got EOF from 127.0.0.1:33552 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 04:39:51.071686 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:51.070521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.19.92.61:45195)
0809 04:39:51.070672 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:51.070677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:51.070757 (+ 80us) server_negotiation.cc:407] Connection header received
0809 04:39:51.070927 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:51.070930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:51.070990 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:51.071067 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:51.071562 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33568: BlockingRecv error: recv got EOF from 127.0.0.1:33568 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 04:39:51.139143 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:51.137911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.19.92.61:45195)
0809 04:39:51.138089 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:51.138093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:51.138233 (+ 140us) server_negotiation.cc:407] Connection header received
0809 04:39:51.138385 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:51.138389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:51.138443 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:51.138519 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:51.139023 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33572: BlockingRecv error: recv got EOF from 127.0.0.1:33572 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 04:39:51.270545 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:51.269306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.19.92.61:45195)
0809 04:39:51.269476 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:51.269479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:51.269633 (+ 154us) server_negotiation.cc:407] Connection header received
0809 04:39:51.269752 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:51.269755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:51.269805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:51.269889 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:51.270408 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33576: BlockingRecv error: recv got EOF from 127.0.0.1:33576 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 04:39:51.530019 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:51.528866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.19.92.61:45195)
0809 04:39:51.529022 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:51.529028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:51.529090 (+ 62us) server_negotiation.cc:407] Connection header received
0809 04:39:51.529265 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:51.529269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:51.529318 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:51.529387 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:51.529887 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33578: BlockingRecv error: recv got EOF from 127.0.0.1:33578 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:39:51.824466 21247 catalog_manager.cc:1259] Loaded cluster ID: 738729a805684589a03beb5b3de730aa
I20260809 04:39:51.824556 21247 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99: loading cluster ID for follower catalog manager: success
I20260809 04:39:51.826290 21247 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99: acquiring CA information for follower catalog manager: success
I20260809 04:39:51.827455 21247 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48a0d28d0ec448f7bd4ffc1fd4f3bd99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:39:51.838181 21266 catalog_manager.cc:1259] Loaded cluster ID: 738729a805684589a03beb5b3de730aa
I20260809 04:39:51.838251 21266 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9: loading cluster ID for follower catalog manager: success
I20260809 04:39:51.839905 21266 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9: acquiring CA information for follower catalog manager: success
I20260809 04:39:51.841045 21266 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04946764b92646ab9f6e1172736219e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:39:52.048007 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:52.043974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.19.92.61:45195)
0809 04:39:52.044215 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:52.044220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:52.044296 (+ 76us) server_negotiation.cc:407] Connection header received
0809 04:39:52.044499 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:52.044502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:52.044634 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:52.044735 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:52.045024 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.045601 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.046486 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.046722 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.046812 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:52.047148 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:52.047612 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:52.047724 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:52.047796 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 04:39:52.047852 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260809 04:39:52.054587 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:52.050992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.19.92.60:33343)
0809 04:39:52.051232 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:52.051236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:52.051250 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:39:52.051362 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:52.051365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:52.051493 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:52.051593 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:52.051803 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.052309 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.053130 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.053407 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.053505 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:52.053789 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:52.054198 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:52.054305 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:52.054380 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 04:39:52.054443 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":63,"threads_started":1}
I20260809 04:39:52.060657 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:52.057223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.19.92.61:45195)
0809 04:39:52.057343 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:52.057346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:52.057490 (+ 144us) server_negotiation.cc:407] Connection header received
0809 04:39:52.057678 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:52.057681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:52.057733 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:52.057778 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:52.058134 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.058783 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.059674 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.059888 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.059984 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:52.060222 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:52.060294 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:52.060352 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:52.060462 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 04:39:52.060518 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 04:39:52.061805 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:52.057653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.19.92.60:33343)
0809 04:39:52.057748 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:52.057751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:52.057977 (+ 226us) server_negotiation.cc:407] Connection header received
0809 04:39:52.058144 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:52.058147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:52.058194 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:52.058263 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:52.058510 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.059287 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.060589 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.060896 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.060965 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:52.061360 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:52.061441 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:52.061519 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:52.061647 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 04:39:52.061697 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 04:39:52.062357 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:52.057275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.19.92.62:36503)
0809 04:39:52.057766 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:52.057771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:52.057854 (+ 83us) server_negotiation.cc:407] Connection header received
0809 04:39:52.058020 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:52.058024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:52.058180 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:52.058284 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:52.058627 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.059410 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.060589 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:52.061195 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:52.061291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:52.061654 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:52.061996 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:52.062086 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:52.062171 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 04:39:52.062223 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":60,"threads_started":1}
I20260809 04:39:52.065104 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 20893
I20260809 04:39:52.079044 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21003
I20260809 04:39:52.090162 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21104
2026-08-09T04:39:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3215 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:39:52 dist-test-slave-rmhg krb5kdc[21414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:39:52 dist-test-slave-rmhg krb5kdc[21414](info): set up 2 sockets
Aug 09 04:39:52 dist-test-slave-rmhg krb5kdc[21414](info): commencing operation
krb5kdc: starting...
W20260809 04:39:53.227818 19824 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/test-user.keytab.
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.92.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
Entry for principal kudu/127.19.92.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab.
I20260809 04:39:53.359695 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36187
--webserver_interface=127.19.92.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36187,127.19.92.61:45975,127.19.92.60:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:39:53.469895 21427 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:39:53.470183 21427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:53.470227 21427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:53.470289 21427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:53.474283 21427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:53.474351 21427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:53.474378 21427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:53.474400 21427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:53.474431 21427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:53.479727 21427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36187,127.19.92.61:45975,127.19.92.60:46231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.62.keytab
--principal=kudu/127.19.92.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.62:36187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.92.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:53.480875 21427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:53.482016 21427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:53.490816 21436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:53.490823 21434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:53.490911 21427 server_base.cc:1034] running on GCE node
W20260809 04:39:53.490816 21433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:53.491678 21427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:53.492833 21427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:53.493990 21427 hybrid_clock.cc:648] HybridClock initialized: now 1786250393493975 us; error 25 us; skew 500 ppm
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:53.501327 21427 init.cc:377] Logged in from keytab as kudu/127.19.92.62@KRBTEST.COM (short username kudu)
I20260809 04:39:53.502249 21427 webserver.cc:460] Webserver started at http://127.19.92.62:33273/ using document root <none> and password file <none>
I20260809 04:39:53.502832 21427 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:53.502904 21427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:53.503100 21427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:53.505254 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/data/instance:
uuid: "8bed0ee978b6407f8baa4f3c19378326"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "9d9e858ed14b893150fee36d93dab02f"
server_key_iv: "20e3b9315dfbb9ffc885d1f131a44617"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.505811 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/wal/instance:
uuid: "8bed0ee978b6407f8baa4f3c19378326"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "9d9e858ed14b893150fee36d93dab02f"
server_key_iv: "20e3b9315dfbb9ffc885d1f131a44617"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.509656 21427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:39:53.512060 21449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:53.513377 21427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:39:53.513491 21427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/wal
uuid: "8bed0ee978b6407f8baa4f3c19378326"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "9d9e858ed14b893150fee36d93dab02f"
server_key_iv: "20e3b9315dfbb9ffc885d1f131a44617"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.513594 21427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:53.538126 21427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:53.541028 21427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:53.541256 21427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:53.548435 21427 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.62:36187
I20260809 04:39:53.548676 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.62:36187 every 8 connection(s)
I20260809 04:39:53.549441 21427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/data/info.pb
I20260809 04:39:53.552150 21521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:53.555832 21521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:53.556752 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21427
I20260809 04:39:53.556870 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-0/wal/instance
I20260809 04:39:53.557189 19824 external_mini_cluster.cc:1428] Setting key b7b4afa4fb61a31b7ad4c947b9f09a05
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.92.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.61@KRBTEST.COM" created.
I20260809 04:39:53.571367 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.558119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.19.92.62:36187)
0809 04:39:53.558611 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:53.558630 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:53.558865 (+ 235us) server_negotiation.cc:407] Connection header received
0809 04:39:53.560066 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:53.560071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:53.560262 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:53.560420 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:53.561447 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.562385 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.563530 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.564111 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.564276 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.566686 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:53.566691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:53.566701 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:53.566723 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:53.568214 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.568775 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.568781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.568782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.568890 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.569274 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.569278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.569280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.569680 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:53.569791 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:53.570251 (+ 460us) server_negotiation.cc:299] Negotiation successful
0809 04:39:53.570473 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":334,"spinlock_wait_cycles":4480,"thread_start_us":133,"threads_started":1}
I20260809 04:39:53.571363 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.558119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36187 (local address 127.0.0.1:58908)
0809 04:39:53.558707 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.558756 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:53.559757 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:53.560469 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:53.560475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:53.560901 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:53.561275 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.561294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.562537 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.562540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.563398 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.563408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.564256 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.564258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.564381 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.565038 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:53.565053 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:53.566544 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:53.568316 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.568324 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.568333 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.568672 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.569000 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.569003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.569006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.569180 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.569806 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:53.569813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:53.569975 (+ 162us) client_negotiation.cc:769] Sending connection context
0809 04:39:53.570157 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 04:39:53.570573 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":137,"spinlock_wait_cycles":50176,"thread_start_us":166,"threads_started":1}
I20260809 04:39:53.574270 21521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:53.575083 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.574776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38520)
0809 04:39:53.574904 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.574999 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 04:39:53.575310 21452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
I20260809 04:39:53.576256 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.576025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38530)
0809 04:39:53.576149 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.576197 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 04:39:53.576458 21521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:45975: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab.
Entry for principal kudu/127.19.92.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab.
I20260809 04:39:53.590112 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.61:45975
--webserver_interface=127.19.92.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab
--principal=kudu/127.19.92.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36187,127.19.92.61:45975,127.19.92.60:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:39:53.625695 21521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } attempt: 1
I20260809 04:39:53.626609 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.626309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38540)
0809 04:39:53.626464 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.626530 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 04:39:53.627343 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.627126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38546)
0809 04:39:53.627237 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.627282 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 04:39:53.627471 21521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:45975: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
I20260809 04:39:53.695643 21521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } attempt: 2
I20260809 04:39:53.696573 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.696235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38554)
0809 04:39:53.696400 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.696493 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 04:39:53.697337 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.697104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38562)
0809 04:39:53.697234 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.697283 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 04:39:53.697456 21521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.61:45975: Network error: Client connection negotiation failed: client connection to 127.19.92.61:45975: connect: Connection refused (error 111)
W20260809 04:39:53.702350 21531 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:39:53.702641 21531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:53.702684 21531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:39:53.702754 21531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:53.706738 21531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:53.706794 21531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:53.706821 21531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:53.706853 21531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:53.706882 21531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:53.712312 21531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36187,127.19.92.61:45975,127.19.92.60:46231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.61.keytab
--principal=kudu/127.19.92.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.61:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.92.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:53.713445 21531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:53.714488 21531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:53.722834 21539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:53.722790 21542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:53.722875 21531 server_base.cc:1034] running on GCE node
W20260809 04:39:53.722788 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:53.723647 21531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:53.724789 21531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:53.725973 21531 hybrid_clock.cc:648] HybridClock initialized: now 1786250393725939 us; error 44 us; skew 500 ppm
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:53.728833 21531 init.cc:377] Logged in from keytab as kudu/127.19.92.61@KRBTEST.COM (short username kudu)
I20260809 04:39:53.729826 21531 webserver.cc:460] Webserver started at http://127.19.92.61:39091/ using document root <none> and password file <none>
I20260809 04:39:53.730382 21531 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:53.730451 21531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:53.730669 21531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:53.732728 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/data/instance:
uuid: "84798872c2da488b9c443aaeec8ba108"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "0b100d31eff3de29aec677adbf42219e"
server_key_iv: "bc24ea84844d881e5cee66818f8f3b7f"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.733322 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/wal/instance:
uuid: "84798872c2da488b9c443aaeec8ba108"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "0b100d31eff3de29aec677adbf42219e"
server_key_iv: "bc24ea84844d881e5cee66818f8f3b7f"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.737133 21531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:39:53.739777 21552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:53.741030 21531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:39:53.741178 21531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/wal
uuid: "84798872c2da488b9c443aaeec8ba108"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "0b100d31eff3de29aec677adbf42219e"
server_key_iv: "bc24ea84844d881e5cee66818f8f3b7f"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.741281 21531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:53.754650 21531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:53.757416 21531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:53.757601 21531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:53.764396 21531 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.61:45975
I20260809 04:39:53.764667 21621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.61:45975 every 8 connection(s)
I20260809 04:39:53.765460 21531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/data/info.pb
I20260809 04:39:53.767459 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21531
I20260809 04:39:53.767598 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-1/wal/instance
I20260809 04:39:53.767879 19824 external_mini_cluster.cc:1428] Setting key 213a271bc5d9f40384ec5d8795680bb4
I20260809 04:39:53.768425 21622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:53.773507 21622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.92.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.92.60@KRBTEST.COM" created.
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:53.788628 21521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } attempt: 3
I20260809 04:39:53.789711 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.776078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.19.92.62:36187)
0809 04:39:53.776257 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:53.776262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:53.777216 (+ 954us) server_negotiation.cc:407] Connection header received
0809 04:39:53.778250 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:53.778253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:53.778305 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:53.778387 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:53.779698 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.780338 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.781890 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.782145 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.782234 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.784882 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:53.784885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:53.784887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:53.784915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:53.786461 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.787163 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.787166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.787168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.787257 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.787678 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.787681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.787682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.787889 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:53.787977 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:53.788764 (+ 787us) server_negotiation.cc:299] Negotiation successful
0809 04:39:53.789550 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:39:53.790263 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.776467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36187 (local address 127.0.0.1:58916)
0809 04:39:53.777008 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.777066 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:53.778037 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:53.778576 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:53.778585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:53.778991 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:53.779535 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.779550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.780489 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.780493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.781760 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.781773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.782312 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.782316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.782526 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.783172 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:53.783189 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:53.784692 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:53.786603 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.786611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.786632 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.787044 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.787354 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.787357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.787363 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.787574 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.788009 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:53.788016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:53.788426 (+ 410us) client_negotiation.cc:769] Sending connection context
0809 04:39:53.788672 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 04:39:53.789066 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":142,"threads_started":1}
I20260809 04:39:53.791707 21622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
I20260809 04:39:53.798861 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.789099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38568)
0809 04:39:53.789326 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.789341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:53.789421 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:53.790158 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:53.790161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:53.790336 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:53.790449 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.790456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.791515 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.791518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.792223 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.792234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.793133 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.793137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.793317 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.793848 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:53.793864 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:53.795261 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:53.797106 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.797109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.797111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.797404 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.797783 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.797786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.797788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.797888 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.798347 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:53.798350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:53.798454 (+ 104us) client_negotiation.cc:769] Sending connection context
0809 04:39:53.798543 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 04:39:53.798690 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260809 04:39:53.798995 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.789256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.19.92.61:45975)
0809 04:39:53.789732 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:53.789746 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:53.789770 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:39:53.789815 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:53.789821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:53.790022 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:53.790156 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:53.790551 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.791400 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.792349 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.793022 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.793312 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.795391 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:53.795400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:53.795412 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:53.795455 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:53.797010 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.797553 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.797559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.797561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.797668 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.797988 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.797991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.797993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.798236 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:53.798361 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:53.798683 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 04:39:53.798824 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"spinlock_wait_cycles":3200,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.92.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab.
Entry for principal kudu/127.19.92.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab.
I20260809 04:39:53.800365 19824 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
/tmp/dist-test-taskWTI4ci/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.60:46231
--webserver_interface=127.19.92.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab
--principal=kudu/127.19.92.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.92.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.92.62:36187,127.19.92.61:45975,127.19.92.60:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:39:53.802115 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.792404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38578)
0809 04:39:53.792539 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.792555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:53.792630 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:53.793188 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:53.793190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:53.793354 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:53.793445 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.793450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.794156 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.794159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.794871 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.794883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.795383 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.795387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.795561 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.796065 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:53.796081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:53.797332 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:53.800766 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.800769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.800771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.801002 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.801278 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.801281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.801282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.801362 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.801724 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:53.801726 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:53.801801 (+ 75us) client_negotiation.cc:769] Sending connection context
0809 04:39:53.801875 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:39:53.802001 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 04:39:53.802481 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.792611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.19.92.61:45975)
0809 04:39:53.792989 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:53.792996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:53.793008 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:39:53.793033 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:53.793035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:53.793068 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:53.793171 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:53.793547 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.794060 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.794998 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.795279 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.795371 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.797475 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:53.797480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:53.797483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:53.797508 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:53.800640 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.801103 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.801106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.801107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.801204 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.801446 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.801449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.801450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.801644 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:53.801737 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:53.801946 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 04:39:53.802062 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":324,"spinlock_wait_cycles":4992,"thread_start_us":184,"threads_started":1}
I20260809 04:39:53.802465 21521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:53.803098 21622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:53.803191 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.802944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38024)
0809 04:39:53.803063 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.803120 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 04:39:53.803714 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.803486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38026)
0809 04:39:53.803594 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.803663 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 04:39:53.803879 21556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
I20260809 04:39:53.804014 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.803708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38040)
0809 04:39:53.803891 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.803938 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260809 04:39:53.804154 21521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:46231: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
I20260809 04:39:53.804934 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.804735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38050)
0809 04:39:53.804832 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.804878 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:39:53.805083 21622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:46231: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
I20260809 04:39:53.835337 21521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } attempt: 1
I20260809 04:39:53.836256 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.835892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38060)
0809 04:39:53.836082 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.836158 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 04:39:53.837003 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.836773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38072)
0809 04:39:53.836876 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.836929 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 04:39:53.837132 21521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:46231: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
I20260809 04:39:53.854365 21622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } attempt: 1
I20260809 04:39:53.855227 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.854861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38082)
0809 04:39:53.855058 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.855134 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 04:39:53.855962 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.855734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38086)
0809 04:39:53.855843 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.855898 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 04:39:53.856115 21622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:46231: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
I20260809 04:39:53.912330 21521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } attempt: 2
I20260809 04:39:53.913201 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.912858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38100)
0809 04:39:53.913025 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.913093 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 04:39:53.914053 21634 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:39:53.914366 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:39:53.914413 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260809 04:39:53.914487 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.914237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38108)
0809 04:39:53.914359 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.914417 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 04:39:53.914511 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:39:53.914628 21521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:46231: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
W20260809 04:39:53.918604 21634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:39:53.918655 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:39:53.918674 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:39:53.918689 21634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:39:53.918713 21634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:39:53.923906 21634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.92.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.92.62:36187,127.19.92.61:45975,127.19.92.60:46231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/krb5kdc/kudu_127.19.92.60.keytab
--principal=kudu/127.19.92.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.92.60:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.92.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 03:57:59 UTC on ccycloud.quasar-dlmoqf.root.comops.site
build id 4575792
I20260809 04:39:53.924274 21622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } attempt: 2
I20260809 04:39:53.925031 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.924730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38124)
0809 04:39:53.924876 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.924940 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 04:39:53.925179 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:39:53.925812 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.925580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38130)
0809 04:39:53.925697 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.925749 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 04:39:53.925961 21622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.92.60:46231: Network error: Client connection negotiation failed: client connection to 127.19.92.60:46231: connect: Connection refused (error 111)
I20260809 04:39:53.926369 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:39:53.934556 21647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:53.934612 21641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:39:53.934621 21642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:39:53.934715 21634 server_base.cc:1034] running on GCE node
I20260809 04:39:53.935295 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:39:53.936333 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:39:53.937500 21634 hybrid_clock.cc:648] HybridClock initialized: now 1786250393937476 us; error 45 us; skew 500 ppm
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:39:53.940047 21634 init.cc:377] Logged in from keytab as kudu/127.19.92.60@KRBTEST.COM (short username kudu)
I20260809 04:39:53.940930 21634 webserver.cc:460] Webserver started at http://127.19.92.60:40113/ using document root <none> and password file <none>
I20260809 04:39:53.941486 21634 fs_manager.cc:359] Metadata directory not provided
I20260809 04:39:53.941548 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:39:53.941720 21634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:39:53.943666 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/data/instance:
uuid: "2dc5119d193840aabafb922e32a19d9e"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "70bcc41c38dcc50160bbf9024cd35010"
server_key_iv: "a6332963d83b7699143612131518c896"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.944202 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/wal/instance:
uuid: "2dc5119d193840aabafb922e32a19d9e"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "70bcc41c38dcc50160bbf9024cd35010"
server_key_iv: "a6332963d83b7699143612131518c896"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.947903 21634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 04:39:53.950325 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:39:53.951508 21634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:39:53.951606 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/data,/tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/wal
uuid: "2dc5119d193840aabafb922e32a19d9e"
format_stamp: "Formatted at 2026-08-09 04:39:53 on dist-test-slave-rmhg"
server_key: "70bcc41c38dcc50160bbf9024cd35010"
server_key_iv: "a6332963d83b7699143612131518c896"
server_key_version: "encryptionkey@0"
I20260809 04:39:53.951689 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:39:53.969847 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:39:53.972584 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:39:53.972746 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:39:53.979624 21634 rpc_server.cc:307] RPC server started. Bound to: 127.19.92.60:46231
I20260809 04:39:53.979887 21738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.92.60:46231 every 8 connection(s)
I20260809 04:39:53.980624 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/data/info.pb
I20260809 04:39:53.983515 21739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:39:53.987484 21739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:53.988464 19824 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu as pid 21634
I20260809 04:39:53.988602 19824 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWTI4ci/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250097044341-19824-0/minicluster-data/master-2/wal/instance
I20260809 04:39:53.988924 19824 external_mini_cluster.cc:1428] Setting key 5a96ee3612f6ef2b4a91d32866f97a3a
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
Aug 09 04:39:53 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
I20260809 04:39:54.001240 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.989665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.19.92.62:36187)
0809 04:39:53.989827 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:53.989832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:53.990785 (+ 953us) server_negotiation.cc:407] Connection header received
0809 04:39:53.991857 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:53.991860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:53.991915 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:53.991992 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:53.992909 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.993470 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.994559 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.994829 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.994905 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.997442 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:53.997446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:53.997449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:53.997495 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:53.998955 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.999558 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.999561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.999562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:53.999651 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:53.999982 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:53.999986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:53.999988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.000206 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.000317 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.000892 (+ 575us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.001045 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:39:54.002162 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.990149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.62:36187 (local address 127.0.0.1:58926)
0809 04:39:53.990631 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:53.990677 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:53.991621 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:53.992060 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:53.992066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:53.992501 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:53.992751 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.992765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.993576 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.993579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.994432 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:53.994442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.994979 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.994983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:53.995153 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.995746 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:53.995763 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:53.997270 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:53.999078 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.999085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.999096 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.999436 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:53.999756 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:53.999758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:53.999759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:53.999888 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.000309 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:54.000314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:54.000638 (+ 324us) client_negotiation.cc:769] Sending connection context
0809 04:39:54.000807 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 04:39:54.001122 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260809 04:39:54.002162 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:53.990924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.19.92.60:46231)
0809 04:39:53.991203 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:53.991213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:53.991259 (+ 46us) server_negotiation.cc:407] Connection header received
0809 04:39:53.991635 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:53.991659 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:53.991908 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:53.992080 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:53.992835 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.993835 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.994734 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:53.995175 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:53.995274 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:53.997892 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:53.997900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:53.997928 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:53.997954 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:53.999608 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.000047 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.000052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.000053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.000122 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.000399 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.000403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.000404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.000688 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.000886 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.001108 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.001257 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260809 04:39:54.004134 21739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:39:54 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
Aug 09 04:39:54 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.62@KRBTEST.COM
I20260809 04:39:54.013007 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.004627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.61:45975 (local address 127.0.0.1:38584)
0809 04:39:54.004797 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:54.004811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:54.004882 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:54.005169 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:54.005172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:54.005338 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:54.005459 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:54.005467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.006399 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.006402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:54.007155 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:54.007165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.007668 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.007671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:54.007751 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.008182 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:54.008197 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:54.009619 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:54.011405 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:54.011445 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:54.011453 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:54.011694 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.011977 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:54.011980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:54.011982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:54.012074 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.012529 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:54.012531 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:54.012629 (+ 98us) client_negotiation.cc:769] Sending connection context
0809 04:39:54.012728 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 04:39:54.012847 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260809 04:39:54.013078 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.004801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.19.92.61:45975)
0809 04:39:54.004921 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.004925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.004940 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:39:54.004986 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.004989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.005037 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.005126 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.005578 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.006288 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.007271 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.007559 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.007658 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.009768 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.009772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.009775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.009804 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.011282 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.011809 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.011811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.011813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.011888 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.012190 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.012194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.012196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.012424 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.012536 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.012792 (+ 256us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.012935 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:39:54.013759 21521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } attempt: 3
I20260809 04:39:54.013985 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.004777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.19.92.62:36187)
0809 04:39:54.004926 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.004929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.004940 (+ 11us) server_negotiation.cc:407] Connection header received
0809 04:39:54.005004 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.005007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.005043 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.005125 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.005825 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.006515 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.007468 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.007749 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.007825 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.010378 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.010382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.010384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.010410 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.012053 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.012535 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.012538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.012539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.012627 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.012924 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.012928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.012930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.013180 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.013381 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.013731 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.013852 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:39:54.014026 21739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:39:54.017069 21622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } attempt: 3
Aug 09 04:39:54 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.62@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
Aug 09 04:39:54 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.60@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
Aug 09 04:39:54 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.92.61@KRBTEST.COM
I20260809 04:39:54.023021 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.014165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38152)
0809 04:39:54.014511 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:54.014524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:54.014598 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:54.014848 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:54.014851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:54.015019 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:54.015122 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:54.015129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.016197 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.016201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:54.016978 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:54.016989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.017600 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.017603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:54.017706 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.018126 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:54.018140 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:54.019322 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:54.021468 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:54.021472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:54.021474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:54.021707 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.021992 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:54.021996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:54.021998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:54.022086 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.022525 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:54.022528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:54.022620 (+ 92us) client_negotiation.cc:769] Sending connection context
0809 04:39:54.022708 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 04:39:54.022841 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277}
I20260809 04:39:54.023252 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.014210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.19.92.60:46231)
0809 04:39:54.014339 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.014344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.014573 (+ 229us) server_negotiation.cc:407] Connection header received
0809 04:39:54.014676 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.014679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.014724 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.015226 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.015327 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.016080 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.017115 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.017479 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.017939 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.019469 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.019473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.019475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.019500 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.021329 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.021816 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.021819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.021821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.021890 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.022186 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.022189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.022190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.022430 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.022512 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.023006 (+ 494us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.023115 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":62}
I20260809 04:39:54.024273 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.014411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38158)
0809 04:39:54.014777 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:54.014790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:54.014845 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:54.015063 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:54.015066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:54.015208 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:54.015310 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:54.015317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.016567 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.016570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:54.017221 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:54.017230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.017744 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.017748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:54.017856 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.018266 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:54.018279 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:54.019822 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:54.021700 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:54.021703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:54.021704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:54.021911 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.022165 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:54.022168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:54.022170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:54.022245 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.023820 (+ 1575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:54.023823 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:54.023922 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 04:39:54.024017 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 04:39:54.024135 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":21}
I20260809 04:39:54.024737 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.014524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.19.92.60:46231)
0809 04:39:54.014843 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.014850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.014864 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:39:54.014909 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.014911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.014954 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.015035 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.015410 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.016471 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.017343 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.017657 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.017742 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.019944 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.019947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.019949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.019972 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.021602 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.022009 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.022012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.022013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.022078 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.022372 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.022376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.022378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.023712 (+ 1334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.023791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.024464 (+ 673us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.024571 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":255,"thread_start_us":142,"threads_started":1}
I20260809 04:39:54.025543 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.015477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.19.92.61:45975)
0809 04:39:54.015578 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.015581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.015727 (+ 146us) server_negotiation.cc:407] Connection header received
0809 04:39:54.015816 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.015819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.015858 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.015920 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.016593 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.018091 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.019095 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.019312 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.019364 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.021971 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.021975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.021978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.022000 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.023601 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.024055 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.024059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.024061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.024141 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.024447 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.024450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.024452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.024649 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.024877 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.025260 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.025391 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 04:39:54.026806 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326: Bootstrap starting.
Aug 09 04:39:54 dist-test-slave-rmhg krb5kdc[21414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250393, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.92.61@KRBTEST.COM for kudu/127.19.92.60@KRBTEST.COM
I20260809 04:39:54.029739 19824 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:39:54.030016 21521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:54.030963 21521 log.cc:824] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:54.031016 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.017426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.92.60:46231 (local address 127.0.0.1:38166)
0809 04:39:54.017576 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 04:39:54.017590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:39:54.017662 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:39:54.021565 (+ 3903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:39:54.021568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:39:54.021732 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:39:54.021830 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:54.021836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.023499 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.023503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:54.024248 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:39:54.024256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.025356 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.025360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:39:54.025457 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.025917 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:39:54.025931 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:39:54.027217 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:39:54.029467 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:54.029470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:54.029472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:54.029712 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.030012 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:39:54.030015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:39:54.030017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:39:54.030111 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:39:54.030549 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:39:54.030552 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:39:54.030646 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 04:39:54.030731 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 04:39:54.030859 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260809 04:39:54.031584 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.017525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.19.92.60:46231)
0809 04:39:54.021249 (+ 3724us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.021261 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.021276 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:39:54.021309 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.021312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.021352 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.021933 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.022044 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.023023 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.024601 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.025240 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.025336 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.027352 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.027356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.027358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.027383 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.029334 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.029790 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.029793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.029795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.029879 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.030230 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.030233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.030235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.030437 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.031242 (+ 805us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.031319 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.031430 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2333,"thread_start_us":112,"threads_started":1}
I20260809 04:39:54.032114 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e: Bootstrap starting.
I20260809 04:39:54.030926 21484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8bed0ee978b6407f8baa4f3c19378326"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58934
I20260809 04:39:54.033517 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326: No bootstrap required, opened a new log
I20260809 04:39:54.034296 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108: Bootstrap starting.
I20260809 04:39:54.034585 21739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:54.035344 21739 log.cc:824] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:54.036434 21521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } }
I20260809 04:39:54.036613 21622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108: Neither blocks nor log segments found. Creating new log.
I20260809 04:39:54.036669 21521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bed0ee978b6407f8baa4f3c19378326, State: Initialized, Role: FOLLOWER
I20260809 04:39:54.037010 21521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } }
I20260809 04:39:54.037480 21622 log.cc:824] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108: Log is configured to *not* fsync() on all Append() calls
I20260809 04:39:54.037528 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e: No bootstrap required, opened a new log
I20260809 04:39:54.037710 21754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.037935 21754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:54.038335 21521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:54.038411 21484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 04:39:54.038525 21484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:54.039649 21484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } }
I20260809 04:39:54.040560 21484 leader_election.cc:288] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [CANDIDATE]: Term 1 election: Requested vote from peers 84798872c2da488b9c443aaeec8ba108 (127.19.92.61:45975), 2dc5119d193840aabafb922e32a19d9e (127.19.92.60:46231)
I20260809 04:39:54.041911 21739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } }
I20260809 04:39:54.042182 21739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc5119d193840aabafb922e32a19d9e, State: Initialized, Role: FOLLOWER
I20260809 04:39:54.042483 21739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } }
W20260809 04:39:54.042781 21456 tablet.cc:2369] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:39:54.043779 21739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [sys.catalog]: configured and running, proceeding with master startup.
W20260809 04:39:54.044454 21766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:54.044534 21766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:54.045499 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bed0ee978b6407f8baa4f3c19378326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2dc5119d193840aabafb922e32a19d9e"
I20260809 04:39:54.045735 21697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:54.043749 21427 master_runner.cc:186] Error getting master registration for 127.19.92.61:45975: OK, instance_id { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" instance_seqno: 1786250393762428 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:39:54.046836 21427 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:39:54.048763 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108: No bootstrap required, opened a new log
I20260809 04:39:54.053372 21622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } }
I20260809 04:39:54.050642 21634 master_runner.cc:186] Error getting master registration for 127.19.92.61:45975: OK, instance_id { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" instance_seqno: 1786250393762428 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:39:54.053732 21634 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:39:54.046993 21697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bed0ee978b6407f8baa4f3c19378326 in term 1.
I20260809 04:39:54.053898 21622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84798872c2da488b9c443aaeec8ba108, State: Initialized, Role: FOLLOWER
I20260809 04:39:54.054179 21452 leader_election.cc:302] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc5119d193840aabafb922e32a19d9e, 8bed0ee978b6407f8baa4f3c19378326; no voters:
I20260809 04:39:54.054491 21754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:39:54.054527 21622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } }
I20260809 04:39:54.054725 21754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [term 1 LEADER]: Becoming Leader. State: Replica: 8bed0ee978b6407f8baa4f3c19378326, State: Running, Role: LEADER
I20260809 04:39:54.055059 21754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } }
I20260809 04:39:54.056222 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.056361 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:54.056730 21622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:39:54.056763 21589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bed0ee978b6407f8baa4f3c19378326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "84798872c2da488b9c443aaeec8ba108"
I20260809 04:39:54.056972 21589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:39:54.056975 21754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bed0ee978b6407f8baa4f3c19378326. Latest consensus state: current_term: 1 leader_uuid: "8bed0ee978b6407f8baa4f3c19378326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.057109 21754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:54.057638 21772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.058336 21589 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bed0ee978b6407f8baa4f3c19378326 in term 1.
I20260809 04:39:54.058481 21772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:54.060714 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.046807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.19.92.60:46231)
0809 04:39:54.046912 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.046916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.049018 (+ 2102us) server_negotiation.cc:407] Connection header received
0809 04:39:54.050164 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.050167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.050213 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.050258 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.052353 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.053170 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.055222 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.055554 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.055629 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.057247 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.057251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.057254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.057280 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.058873 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.059347 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.059351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.059353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.059435 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.059732 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.059736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.059738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.059959 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.060079 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.060450 (+ 371us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.060571 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 04:39:54.062876 21791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:54.062943 21791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:54.066428 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.045314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.19.92.62:36187)
0809 04:39:54.045427 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.045431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.057645 (+ 12214us) server_negotiation.cc:407] Connection header received
0809 04:39:54.057843 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.057846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.057898 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.057948 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.058772 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.059500 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.060494 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.060806 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.060880 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.062558 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.062562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.062565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.062592 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.064235 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.064775 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.064779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.064781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.064867 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.065206 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.065210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.065212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.065453 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.065574 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.065923 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.066041 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:39:54.066888 21800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:39:54.067845 21800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:39:54.069705 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:54.046410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.19.92.61:45975)
0809 04:39:54.046524 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.046528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.047593 (+ 1065us) server_negotiation.cc:407] Connection header received
0809 04:39:54.047770 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.047773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.047817 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.047864 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.049285 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.050025 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.054290 (+ 4265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:54.054615 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:54.054694 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:54.065853 (+ 11159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:39:54.065856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:39:54.065860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:39:54.065885 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:39:54.067875 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.068332 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.068335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.068337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.068437 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:39:54.068759 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:39:54.068762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:39:54.068764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:39:54.068983 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:39:54.069112 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:54.069440 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 04:39:54.069562 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 04:39:54.072454 21812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:39:54.072542 21812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:39:54.076656 21697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [term 1 FOLLOWER]: Refusing update from remote peer 8bed0ee978b6407f8baa4f3c19378326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:54.076712 21589 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [term 1 FOLLOWER]: Refusing update from remote peer 8bed0ee978b6407f8baa4f3c19378326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:39:54.077706 21796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:54.077944 21754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:39:54.081772 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bed0ee978b6407f8baa4f3c19378326. Latest consensus state: current_term: 1 leader_uuid: "8bed0ee978b6407f8baa4f3c19378326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.081794 21772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bed0ee978b6407f8baa4f3c19378326. Latest consensus state: current_term: 1 leader_uuid: "8bed0ee978b6407f8baa4f3c19378326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.081887 21772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:54.081887 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:54.087740 21754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bed0ee978b6407f8baa4f3c19378326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.087869 21754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:54.089354 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bed0ee978b6407f8baa4f3c19378326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.089468 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:54.090152 21815 mvcc.cc:205] Tried to move back new op lower bound from 7316481614130769920 to 7316481614053322752. Current Snapshot: MvccSnapshot[applied={T|T < 7316481614130769920}]
I20260809 04:39:54.090703 21817 mvcc.cc:205] Tried to move back new op lower bound from 7316481614130769920 to 7316481614053322752. Current Snapshot: MvccSnapshot[applied={T|T < 7316481614130769920}]
I20260809 04:39:54.094219 21800 catalog_manager.cc:1347] Generated new cluster ID: 20827b9633d24c22872ca9f079424672
I20260809 04:39:54.094296 21800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:39:54.101120 21796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bed0ee978b6407f8baa4f3c19378326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.101284 21796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326 [sys.catalog]: This master's current role is: LEADER
I20260809 04:39:54.102675 21772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bed0ee978b6407f8baa4f3c19378326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed0ee978b6407f8baa4f3c19378326" member_type: VOTER last_known_addr { host: "127.19.92.62" port: 36187 } } peers { permanent_uuid: "84798872c2da488b9c443aaeec8ba108" member_type: VOTER last_known_addr { host: "127.19.92.61" port: 45975 } } peers { permanent_uuid: "2dc5119d193840aabafb922e32a19d9e" member_type: VOTER last_known_addr { host: "127.19.92.60" port: 46231 } } }
I20260809 04:39:54.102792 21772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:39:54.103534 21818 mvcc.cc:205] Tried to move back new op lower bound from 7316481614130769920 to 7316481614053322752. Current Snapshot: MvccSnapshot[applied={T|T < 7316481614130769920}]
I20260809 04:39:54.138406 21800 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:39:54.139878 21800 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:39:54.153609 21800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bed0ee978b6407f8baa4f3c19378326: Generated new TSK 0
I20260809 04:39:54.155716 21800 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:39:54.185521 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.182383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.19.92.61:45975)
0809 04:39:54.182509 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.182513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.184373 (+ 1860us) server_negotiation.cc:407] Connection header received
0809 04:39:54.184605 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.184609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.184659 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.184711 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.185385 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38610: BlockingRecv error: recv got EOF from 127.0.0.1:38610 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 04:39:54.186854 21749 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.183100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.19.92.60:46231)
0809 04:39:54.183212 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.183217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.186009 (+ 2792us) server_negotiation.cc:407] Connection header received
0809 04:39:54.186169 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.186172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.186221 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.186269 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.186734 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38196: BlockingRecv error: recv got EOF from 127.0.0.1:38196 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 04:39:54.188356 21523 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.183764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.19.92.62:36187)
0809 04:39:54.183874 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.183878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.187243 (+ 3365us) server_negotiation.cc:407] Connection header received
0809 04:39:54.187608 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.187611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.187665 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.187714 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.188239 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58948: BlockingRecv error: recv got EOF from 127.0.0.1:58948 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 04:39:54.195109 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.193531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.19.92.61:45975)
0809 04:39:54.193633 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.193637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.194136 (+ 499us) server_negotiation.cc:407] Connection header received
0809 04:39:54.194340 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.194343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.194419 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.194474 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.195003 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38626: BlockingRecv error: recv got EOF from 127.0.0.1:38626 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260809 04:39:54.200201 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.198705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.19.92.61:45975)
0809 04:39:54.198807 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.198811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.199317 (+ 506us) server_negotiation.cc:407] Connection header received
0809 04:39:54.199500 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.199503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.199554 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.199603 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.200107 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38640: BlockingRecv error: recv got EOF from 127.0.0.1:38640 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 04:39:54.205890 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.204324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.19.92.61:45975)
0809 04:39:54.204454 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.204457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.204948 (+ 491us) server_negotiation.cc:407] Connection header received
0809 04:39:54.205125 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.205128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.205262 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.205313 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.205791 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38652: BlockingRecv error: recv got EOF from 127.0.0.1:38652 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260809 04:39:54.213706 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.212122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.19.92.61:45975)
0809 04:39:54.212255 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.212259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.212757 (+ 498us) server_negotiation.cc:407] Connection header received
0809 04:39:54.212963 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.212967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.213019 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.213070 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.213602 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38666: BlockingRecv error: recv got EOF from 127.0.0.1:38666 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 04:39:54.225950 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.224124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.19.92.61:45975)
0809 04:39:54.224239 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.224244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.224843 (+ 599us) server_negotiation.cc:407] Connection header received
0809 04:39:54.225079 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.225083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.225141 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.225234 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.225821 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38676: BlockingRecv error: recv got EOF from 127.0.0.1:38676 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 04:39:54.249585 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.247713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.19.92.61:45975)
0809 04:39:54.247834 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.247839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.248447 (+ 608us) server_negotiation.cc:407] Connection header received
0809 04:39:54.248697 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.248702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.248769 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.248827 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.249463 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38690: BlockingRecv error: recv got EOF from 127.0.0.1:38690 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 04:39:54.285449 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.284137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.19.92.61:45975)
0809 04:39:54.284312 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.284316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.284398 (+ 82us) server_negotiation.cc:407] Connection header received
0809 04:39:54.284599 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.284602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.284660 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.284743 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.285314 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38694: BlockingRecv error: recv got EOF from 127.0.0.1:38694 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260809 04:39:54.352538 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.351499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.19.92.61:45975)
0809 04:39:54.351656 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.351659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.351753 (+ 94us) server_negotiation.cc:407] Connection header received
0809 04:39:54.351880 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.351883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.351933 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.352009 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.352391 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38710: BlockingRecv error: recv got EOF from 127.0.0.1:38710 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260809 04:39:54.483947 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.482768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.19.92.61:45975)
0809 04:39:54.482922 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.482927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.483036 (+ 109us) server_negotiation.cc:407] Connection header received
0809 04:39:54.483198 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.483201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.483250 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.483320 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.483820 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38720: BlockingRecv error: recv got EOF from 127.0.0.1:38720 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 04:39:54.743360 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:39:54.742264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.19.92.61:45975)
0809 04:39:54.742382 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:54.742386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:54.742423 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:39:54.742595 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:54.742597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:54.742645 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:54.742713 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:39:54.743239 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38734: BlockingRecv error: recv got EOF from 127.0.0.1:38734 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 04:39:55.064265 21791 catalog_manager.cc:1259] Loaded cluster ID: 20827b9633d24c22872ca9f079424672
I20260809 04:39:55.064350 21791 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e: loading cluster ID for follower catalog manager: success
I20260809 04:39:55.066089 21791 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e: acquiring CA information for follower catalog manager: success
I20260809 04:39:55.067260 21791 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2dc5119d193840aabafb922e32a19d9e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:39:55.073850 21812 catalog_manager.cc:1259] Loaded cluster ID: 20827b9633d24c22872ca9f079424672
I20260809 04:39:55.073935 21812 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108: loading cluster ID for follower catalog manager: success
I20260809 04:39:55.075551 21812 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108: acquiring CA information for follower catalog manager: success
I20260809 04:39:55.076687 21812 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84798872c2da488b9c443aaeec8ba108: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:39:55.261669 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:55.257473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.19.92.61:45975)
0809 04:39:55.257719 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:55.257724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:55.257798 (+ 74us) server_negotiation.cc:407] Connection header received
0809 04:39:55.257959 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:55.257962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:55.258092 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:55.258218 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:55.258507 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.259087 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.260023 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.260256 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.260347 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:55.260669 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:55.261266 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:55.261381 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:55.261446 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 04:39:55.261514 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260809 04:39:55.268401 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:55.264765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.19.92.60:46231)
0809 04:39:55.264997 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:55.265001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:55.265067 (+ 66us) server_negotiation.cc:407] Connection header received
0809 04:39:55.265234 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:55.265236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:55.265356 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:55.265463 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:55.265671 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.266189 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.267026 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.267236 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.267305 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:55.267604 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:55.268042 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:55.268131 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:55.268198 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 04:39:55.268247 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":60,"threads_started":1}
I20260809 04:39:55.275705 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:55.271425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.19.92.61:45975)
0809 04:39:55.271548 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:55.271552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:55.272024 (+ 472us) server_negotiation.cc:407] Connection header received
0809 04:39:55.272129 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:55.272132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:55.272177 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:55.272237 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:55.272452 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.273205 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.274569 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.274877 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.274953 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:55.275223 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:55.275303 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:55.275362 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:55.275517 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 04:39:55.275568 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:39:55.276167 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:55.271792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.19.92.60:46231)
0809 04:39:55.271899 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:55.271903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:55.272130 (+ 227us) server_negotiation.cc:407] Connection header received
0809 04:39:55.272177 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:55.272179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:55.272223 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:55.272282 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:55.272680 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.273432 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.275011 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.275251 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.275349 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:55.275602 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:55.275685 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:55.275755 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:55.276017 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 04:39:55.276072 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 04:39:55.276201 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:39:55.271391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.19.92.62:36187)
0809 04:39:55.271685 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0809 04:39:55.271689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:39:55.271774 (+ 85us) server_negotiation.cc:407] Connection header received
0809 04:39:55.272004 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:39:55.272007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:39:55.272180 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:39:55.272293 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:39:55.272714 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.273304 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.274245 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:39:55.274478 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:39:55.274566 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:39:55.274982 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:39:55.275344 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:39:55.275428 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:39:55.275764 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 04:39:55.276058 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":116,"threads_started":1}
I20260809 04:39:55.278981 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21427
I20260809 04:39:55.300395 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21531
I20260809 04:39:55.314383 19824 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWTI4ci/build/debug/bin/kudu with pid 21634
2026-08-09T04:39:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3230 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6445 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298270 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST